An open source design system that's fully customizable and agent ready
TypeScript
feat(cli): self-check in the agent-init prompt to stop CSS/Tailwind drift (#4044)
Joeyc899736a94de
diff --git a/internal/vibe-tests/prompt-purity-test/.gitignore b/internal/vibe-tests/prompt-purity-test/.gitignorenew file mode 100644index 0000000000..cdba529df3--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/.gitignore@@ -0,0 +1,8 @@+# Runtime artifacts default to os.tmpdir()/astryx-purity (out of the repo).+# These patterns only matter if you pass --out pointing inside the repo.+sandbox/+results/+runs/+purity-config-*.json+purity-summary-*.json+*.logdiff --git a/internal/vibe-tests/prompt-purity-test/ITERATE.md b/internal/vibe-tests/prompt-purity-test/ITERATE.mdnew file mode 100644index 0000000000..dec3920caf--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/ITERATE.md@@ -0,0 +1,141 @@+# ITERATE — autonomous prompt-purity search (driven by `/loop`)++This is the per-tick playbook for **automatically evolving the astryx-init prompt** until+agents reliably stop veering into raw CSS/Tailwind (or catch themselves when they do).+It is meant to be run by the `/loop` command, self-paced (no fixed interval), gated on+each harness run completing.++**Invocation** (from the repo root, in a live Cursor agent session):++```+/loop follow internal/vibe-tests/prompt-purity-test/ITERATE.md+```++No interval => dynamic mode: run one iteration now, then wake when the current harness+run finishes and run the next. This is FULLY AUTONOMOUS with NO iteration cap — it stops+only on a confirmed winner or a genuine plateau (see Stop conditions). Token spend is+accepted: a better shipped prompt saves far more downstream.++Prereq (once): the `cursor-agent` CLI must be logged in (`cursor-agent status`). If it+is not, stop and tell the user; do not loop.++---++## State (survives across ticks)++Keep a running log so each tick can compare against every prior attempt and never+re-test a wording:++- Path: `"$TMPDIR/astryx-purity/_loop/iterations-log.json"` (outside the repo, like all+ runtime artifacts). Create it on the first tick if absent: `{ "tick": 0, "champion":+ "B-selfcheck", "history": [] }`.+- Each `history[]` entry: `{ tick, expId, ranConditions, variantHashes, rates: { <cond>:+ { veeredUncaught, caughtGivenVeered, finalPurityMean, n } }, note }`.+- "champion" = the best variant found so far (lowest `veeredUncaught`, tie-broken by+ higher `finalPurityMean`). Seed champion = `B-selfcheck`.++`A-control` is the immutable baseline — never edit `variants/A-control.md`.++---++## One tick++### 1. Read where we are+- Read `iterations-log.json`. Identify the current `champion`.+- If `history` is empty, this is the seed tick: run `A-control`, `B-selfcheck`,+ `C-strong` together (exploration settings below) to establish the baseline and pick+ the first champion. Skip step 2 on the seed tick.++### 2. Author ONE new challenger (skip on the seed tick)+- Based on the last run's per-run detail in `purity-summary.json` (which markers survived+ in `veeredUncaught` runs, whether `ranDocRetrieval` was low, whether catches happened+ late), write ONE new variant wording that targets the observed failure mode. Ideas to+ vary (change one axis at a time so you learn what matters):+ - imperative strength / ordering (self-check first vs last)+ - explicit marker checklist vs terse rule+ - mandating `astryx component`/`search` verification vs not+ - "re-read the file" framing vs "before you output" framing+ - length (a terse one-liner sometimes beats a wall of text)+- Write it to a NEW file `variants/g<NN>-<slug>.md` (never overwrite A/B/C or a prior+ generation — history must stay reproducible). Add a matching entry to+ `conditions.json` `conditions[]`: `{ "id": "g<NN>-<slug>", "role": "candidate",+ "variant": "g<NN>-<slug>" }`.+- Do NOT test a wording whose text hash already appears in `history`.++### 3. Run (exploration = cheap, fast)+- Compare exactly THREE conditions so runs stay focused: control, champion, challenger.+- Explore settings: `--prompts dd-1,tc-6 --reps 3`.+- Start it in the BACKGROUND and arm the completion wake (dynamic `/loop`). Generate an+ id first so the wake payload carries it:++```bash+EXP=$(node -e "console.log(require('crypto').randomBytes(4).toString('hex'))")+( node internal/vibe-tests/prompt-purity-test/run-purity.mjs \+ --exp "$EXP" --conditions A-control,<champion>,<challenger> --prompts dd-1,tc-6 --reps 3 \+ --concurrency 6 ; \+ echo "AGENT_LOOP_WAKE_purity {\"expId\":\"$EXP\",\"phase\":\"explore\"}" ) 2>&1+```++ Launch via the Shell tool with `block_until_ms: 0` and `notify_on_output` on+ `^AGENT_LOOP_WAKE_purity`. Also arm ONE long fallback heartbeat (e.g. `sleep 1800`)+ per the loop skill. Then END THE TURN — do not block.++### 4. On wake — evaluate+- Read `"$TMPDIR/astryx-purity/$EXP/purity-summary.json"`.+- Append a `history` entry (rates for each condition + variant hashes).+- Compare challenger vs champion: challenger becomes the new champion if its+ `veeredUncaught` mean is lower (tie-break higher `finalPurityMean`). Update `champion`.+- Report a one-line tick summary to the user: control vs champion vs challenger on+ `veeredUncaught`, `caughtGivenVeered`, purity — and whether the champion changed.++### 5. Decide: confirm, continue, or stop+- **Promote to confirmation** when the champion beats `A-control` in exploration+ (`veeredUncaught` lower AND purity higher). Run a CONFIRMATION:+ `--conditions A-control,<champion> --prompts dd-1,dd-5,wd-1,wd-4,tc-6,ps-1 --reps 7`+ (K>=5, full curated set), same background+wake pattern with `"phase":"confirm"`.+- On a confirmation wake: if the champion is **decisive** — its `veeredUncaught` Wilson+ upper bound is below control's lower bound AND mean purity is higher — go to Ship.+ Otherwise record it, keep searching (back to step 2).+- **Continue** (no interval): re-arm and start the next tick (step 2).++---++## Stop conditions++Stop the loop (kill any watcher/heartbeat PID, do not re-arm) when ANY holds:++- **Winner confirmed** — a champion passed the confirmation run decisively. Go to Ship.+- **Plateau** — the best `veeredUncaught` has not improved across the last 3 distinct+ challengers. Report the best-so-far; ship only if it also beats control decisively,+ else report that no variant reliably beats control and hand back to the user.+- **Floor** — `A-control` already shows ~0 `veeredUncaught` on the confirmation set+ (nothing to fix); report and stop.+- The user says stop.++---++## Ship (only after a confirmed winner)++This completes Phase 4 / the `ship` to-do:++1. Bake the winning wording into `generateCompressedIndex()` in+ `packages/cli/src/commands/agent-docs.mjs` — add the self-check lines to the emitted+ block (respect the `stylex` / `tailwind` / `css` styling-system branches).+2. Update the test `packages/cli/src/commands/agent-docs.test.mjs` to assert the new+ self-check text is present in the generated index.+3. Run the CLI tests for that file.+4. Report: the winning variant text, its confirmation rates vs control, and the diff.+5. Stop the loop.++---++## Reporting & safety++- Every tick: emit a short status (which conditions ran, the funnel deltas, champion+ change, next action). On stop: say why and give the final winner + rates.+- All runs stay under `"$TMPDIR/astryx-purity"` (never the repo), so parallel Cursor+ work, file-watchers, and `git status` are untouched. Only `conditions.json` ++ `variants/g*.md` (and, on Ship, the two CLI files) change in the repo.+- Never edit `variants/A-control.md`. Never place `expectedComponents` in any prompt.+- One loop only — before arming, check for an already-running `AGENT_LOOP_WAKE_purity`+ watcher and reuse it.diff --git a/internal/vibe-tests/prompt-purity-test/PLAN.md b/internal/vibe-tests/prompt-purity-test/PLAN.mdnew file mode 100644index 0000000000..c236de8eb5--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/PLAN.md@@ -0,0 +1,138 @@+# Prompt Purity Vibe Test++**Status:** implemented (harness); iterating on variants+**Related:** `../cli-discovery-test/PLAN.md` (blueprint), `../src/universal-eval.ts` (reused detectors), `packages/cli/src/commands/agent-docs.mjs` (the prompt under test)++---++## 1. The question++Agents are already decent at using Astryx — but on hard tasks they **veer** into raw+CSS / Tailwind / inline styles / hardcoded values instead of Astryx components ++`xstyle`/tokens. We want to change the **injected init prompt** so the agent+**self-verifies** its output and **catches itself** before finishing.++> Does adding a post-generation self-check to the `astryx init` prompt make agents+> reliably (a) *not* leave raw CSS/Tailwind in the final file, and (b) when they do+> start to veer, **catch and correct it** — measured from the actual run trajectory?++## 2. What varies (independent variable)++The text that `astryx init --features agents` splices between `<!-- ASTRYX:START -->`+markers, generated by `generateCompressedIndex()` in+`packages/cli/src/commands/agent-docs.mjs`. We do **not** edit that source during+experimentation. Instead each condition = a real `astryx init` AGENTS.md + one+**variant** self-check block spliced inside the markers (`variants/*.md`,+`conditions.json`). Only the injected prompt varies; environment, prompts, and model+are held constant (Checker Protocol §2). The winning variant is baked into+`generateCompressedIndex()` only at the end.++## 3. What we measure (dependent variables)++Per run, from the **purity timeline** — the ordered snapshots of the output `.tsx`+captured live during the run (`run-purity.ts` via `fs.watch` + a poll fallback),+classified in `purity-eval.ts`:++- `neverVeered` — no non-Astryx styling marker ever appeared. (best)+- `caught` — a marker appeared at some step but the **final** file is clean. (the+ self-check working: veered, then caught)+- `veeredUncaught` — a marker is still present at the end. (the bad outcome to minimize)+- `finalPurity` — 0–100 graded score of the final file.++**Veer markers** (see `purity-eval.ts`, tunable in one place): `className=`,+`style={{…}}`, `.css` import / `@apply`, Tailwind utility classes, hardcoded+`#hex`/`rgb()` colors (hard markers → drive the binary "veered"); hardcoded+`px`/`rem`, styled raw elements (soft markers → affect the score only).++Secondary signals: `ranAstryxSelfCheck` (did it run `astryx search/component/docs`+to verify — from the logging shim), and a transcript scan for "caught-in-reasoning"+(proposed CSS then reconsidered before writing).++## 4. Harness++- `setup-purity.mjs` — builds one isolated project per `condition × prompt × rep`+ **under an out-of-workspace sandbox** (`os.tmpdir()/astryx-purity/<expId>` by+ default; `--out` to override). Bare base = `@astryxdesign/core` + theme symlinked,+ a **StyleX compiler plugin listed in `package.json`** so the CLI detects the+ `stylex` styling system (the Astryx path), a logging `astryx` shim, then real+ `astryx init` + the variant splice. Writes `tasks/<id>.json` + `purity-config-<expId>.json`.+- `run-purity.ts` — runner. Shells out to the `cursor-agent` CLI headless (`--print+ --output-format stream-json --model claude-opus-4-8-max-fast --force --trust+ --workspace <sandbox project>`), snapshots the `.tsx` on every change, persists the+ streamed events to `transcript.jsonl`, records runtime facts to `run.json`. Bounded+ concurrency (`--concurrency`, default 3) + per-run timeout so it never starves+ interactive Cursor usage.+- `purity-eval.ts` — veer-marker detectors + timeline classifier (also reuses+ `evaluate()` from `../src/universal-eval` for a supplementary graded score).+- `purity-aggregate.ts` — per-condition Wilson 95% CI rates + comparison vs control;+ writes `purity-summary-<expId>.json`.+- `run-purity.mjs` — one-command orchestrator (setup → run → aggregate); the unit+ `/loop` would drive.+- `smoke-test.ts` — **no-LLM** verification of the classifier + setup wiring.++## 5. Isolation (never interferes with parallel Cursor work)++- All runtime artifacts live under an **out-of-workspace sandbox** (temp dir /+ worktree), never in the open repo, so file-watchers, other Cursor agents, and+ `git status` are untouched.+- Each agent runs with `cwd` = its sandbox project and `settingSources: []`; repo+ packages are only **read** via symlink. `fs.watch` watches only the sandbox.+- Uses the `cursor-agent` CLI scoped to the sandbox via `--workspace`; bounded+ concurrency + per-run timeout keep interactive Cursor usage responsive. No separate+ API key needed — it uses your existing `cursor-agent` login.++## 6. Procedure++One prerequisite for LIVE runs (the no-LLM smoke test and `--dry-run` need nothing):+the `cursor-agent` CLI must be logged in.++```bash+cursor-agent status # should say "Logged in as ..."; if not: cursor-agent login+```++```bash+# no-LLM harness check (zero model calls) — always safe to run+npx tsx prompt-purity-test/smoke-test.ts++# one-shot: setup -> run -> aggregate (pilot default: dd-1, tc-6; A vs B; K=3)+node prompt-purity-test/run-purity.mjs --conditions A-control,B-selfcheck --prompts dd-1,tc-6 --reps 3++# or step by step+node prompt-purity-test/setup-purity.mjs --exp <id> --conditions A-control,B-selfcheck --prompts dd-1,tc-6 --reps 3+npx tsx prompt-purity-test/run-purity.ts --experiment <id> --concurrency 3+npx tsx prompt-purity-test/purity-aggregate.ts --experiment <id>+```++## 7. Decision rule++A variant **wins** when its `veeredUncaught` rate is significantly below control+(its upper Wilson bound clears control's lower bound) and/or `caught | veered` and+mean `finalPurity` are significantly higher, holding across the curated hard prompts+at K ≥ 5. Then bake it into `generateCompressedIndex()`.++## 8. Checker Protocol compliance++- §2 only the SUT varies — one bare base, single-variant splice, identical prompts/env/model.+- §3 never leak the answer — `expectedComponents` is eval-only, never in the task prompt.+- §4 representative environment — realistic "installed @astryxdesign/core" project.+- §5 context-free agents — fresh SDK agent per run, `settingSources: []`, K independent reps.++## 9. Autonomous iteration (`/loop`)++Once the prereqs are in place, the whole search runs itself. In a live Cursor agent+session:++```+/loop follow internal/vibe-tests/prompt-purity-test/ITERATE.md+```++Dynamic (self-paced) mode: it runs one iteration now, then wakes when each harness run+finishes (event-gated on the orchestrator's completion line) and runs the next. Per tick+it authors ONE new challenger variant aimed at the last run's failure mode, compares+control vs champion vs challenger cheaply (K=3 on `dd-1,tc-6`), promotes the best, then+confirms a leader at K>=5 on the full curated set. It is fully autonomous with NO+iteration cap — it stops only on a confirmed winner (then bakes the wording into+`generateCompressedIndex()` and updates the CLI test — Phase 4) or a genuine plateau.+Cross-tick memory lives in `"$TMPDIR/astryx-purity/_loop/iterations-log.json"`; all runs+stay in the tmpdir sandbox, so your parallel Cursor work is never touched. See+`ITERATE.md` for the full per-tick procedure and stop conditions.diff --git a/internal/vibe-tests/prompt-purity-test/conditions.json b/internal/vibe-tests/prompt-purity-test/conditions.jsonnew file mode 100644index 0000000000..2554368160--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/conditions.json@@ -0,0 +1,28 @@+{+ "name": "prompt-purity",+ "description": "A/B the astryx-init injected prompt (generateCompressedIndex in packages/cli/src/commands/agent-docs.mjs). Each condition = a fresh `astryx init` AGENTS.md + one variant self-check block spliced inside the <!-- ASTRYX:START/END --> markers. Only the injected prompt varies; environment, prompts, and model are held constant (Checker Protocol §2).",+ "control": "A-control",+ "conditions": [+ {+ "id": "A-control",+ "label": "Injected prompt as-is (no self-check)",+ "role": "control",+ "variant": "A-control",+ "hypothesis": "On hard tasks the agent veers into raw CSS/Tailwind and leaves it in the final file."+ },+ {+ "id": "B-selfcheck",+ "label": "+ post-generation self-check block",+ "role": "candidate",+ "variant": "B-selfcheck",+ "hypothesis": "A re-read-and-fix step makes the agent catch veers it introduced, raising final purity."+ },+ {+ "id": "C-strong",+ "label": "+ stronger imperative self-check with CLI verify",+ "role": "candidate",+ "variant": "C-strong",+ "hypothesis": "A more forceful, checklist-style self-check with a mandated `astryx component` verify catches more veers, at some token cost."+ }+ ]+}diff --git a/internal/vibe-tests/prompt-purity-test/prompts.json b/internal/vibe-tests/prompt-purity-test/prompts.jsonnew file mode 100644index 0000000000..aab0d25fb2--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/prompts.json@@ -0,0 +1,20 @@+{+ "name": "prompt-purity-prompts",+ "description": "Curated veer-prone subset referenced BY ID from ../test-sets/default.json. Hard, layout/data/theming-heavy tasks are where agents abandon the design system for raw CSS/Tailwind. expectedComponents is eval-only and is NEVER shown to the agent (Checker Protocol §3). Referenced by id to avoid answer-leak and drift.",+ "source": "../test-sets/default.json",+ "selectionStrategy": "hardest, most veer-prone tasks across distinct categories; K>=5 reps per prompt per condition since veering is stochastic.",+ "promptIds": [+ {"id": "dd-1", "category": "data-display", "complexity": "complex"},+ {"id": "dd-5", "category": "data-display", "complexity": "complex"},+ {"id": "wd-1", "category": "workflow-description", "complexity": "complex"},+ {"id": "wd-4", "category": "workflow-description", "complexity": "complex"},+ {"id": "tc-6", "category": "theme-customization", "complexity": "complex"},+ {"id": "ps-1", "category": "page-setup", "complexity": "moderate"}+ ],+ "pilotDefault": ["dd-1", "tc-6"],+ "notes": [+ "tc-6 explicitly asks for the XDS theme system 'not inline styles' — a direct veer probe.",+ "dd-1/dd-5 (tables) and wd-4 (a full internal tool) strongly tempt hand-rolled markup + CSS.",+ "IDs verified against ../test-sets/default.json. Do NOT copy prompt text or expectedComponents here."+ ]+}diff --git a/internal/vibe-tests/prompt-purity-test/purity-aggregate.ts b/internal/vibe-tests/prompt-purity-test/purity-aggregate.tsnew file mode 100644index 0000000000..8494461e5e--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/purity-aggregate.ts@@ -0,0 +1,423 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file purity-aggregate.ts+ * @input --experiment <id> [--out <base>] (reads <sandbox>/<id>/purity-config.json)+ * @output Per-condition purity funnel with Wilson 95% CIs + comparison vs control; writes purity-summary.json+ * @position internal/vibe-tests/prompt-purity-test — scores the prompt-purity experiment+ *+ * Reconstructs each run's purity timeline from its captured snapshots (the objective+ * record of what the agent actually wrote, moment to moment), classifies veered/caught+ * via purity-eval.ts, and reports per-condition rates with Wilson 95% CIs so "reliably+ * catches its veers" is legible. Ground-truth CLI usage comes from the shim log.+ *+ * Usage: npx tsx purity-aggregate.ts --experiment <id> [--out <base>]+ */++import * as fs from 'node:fs';+import * as os from 'node:os';+import * as path from 'node:path';+import {+ classifyTimeline,+ parseResultEvent,+ estimateCostUSD,+ emptyUsage,+ addUsage,+ type Classification,+ type TokenUsage,+} from './purity-eval.js';++const DOC_RETRIEVAL_CMDS = new Set(['component', 'build', 'search', 'template', 'docs', 'swizzle']);++function sandboxBase(out?: string): string {+ return out || process.env.ASTRYX_PURITY_OUT || path.join(os.tmpdir(), 'astryx-purity');+}++/** Wilson score interval for a binomial proportion (95% by default). */+function wilson(successes: number, n: number, z = 1.96): {p: number; lo: number; hi: number} {+ if (n === 0) {return {p: 0, lo: 0, hi: 0};}+ const phat = successes / n;+ const z2 = z * z;+ const denom = 1 + z2 / n;+ const center = phat + z2 / (2 * n);+ const margin = z * Math.sqrt((phat * (1 - phat) + z2 / (4 * n)) / n);+ return {p: phat, lo: Math.max(0, (center - margin) / denom), hi: Math.min(1, (center + margin) / denom)};+}++interface Rate {+ count: number;+ n: number;+ pct: number;+ lo: number;+ hi: number;+}+function rate<T>(items: T[], pick: (x: T) => boolean): Rate {+ const s = items.filter(pick).length;+ const {p, lo, hi} = wilson(s, items.length);+ return {count: s, n: items.length, pct: p, lo, hi};+}+const fmtRate = (r: Rate) => `${(r.pct * 100).toFixed(0)}% [${(r.lo * 100).toFixed(0)}-${(r.hi * 100).toFixed(0)}] (${r.count}/${r.n})`;++interface RunResult {+ taskId: string;+ basePromptId: string;+ category: string;+ condition: string;+ status: string;+ model: string;+ completed: boolean;+ ranAstryxCli: boolean;+ ranDocRetrieval: boolean;+ cliCommands: string[];+ usage: TokenUsage | null;+ estCostUSD: number | null;+ apiDurationMs: number | null;+ cls: Classification;+}++function readSnapshots(runDir: string): string[] {+ const snapDir = path.join(runDir, 'snapshots');+ if (!fs.existsSync(snapDir)) {return [];}+ return fs+ .readdirSync(snapDir)+ .filter(f => f.endsWith('.tsx'))+ .sort() // NNNN-<ts> prefix sorts chronologically+ .map(f => fs.readFileSync(path.join(snapDir, f), 'utf-8'));+}++function readTranscriptText(runDir: string, cap = 200_000): string {+ const p = path.join(runDir, 'transcript.jsonl');+ if (!fs.existsSync(p)) {return '';}+ let text = '';+ for (const line of fs.readFileSync(p, 'utf-8').split('\n')) {+ if (!line.trim()) {continue;}+ try {+ const ev = JSON.parse(line) as {message?: {content?: unknown}; text?: unknown};+ const content = ev?.message?.content;+ if (Array.isArray(content)) {+ for (const block of content as Array<{type?: string; text?: string}>) {+ if (block?.type === 'text' && typeof block.text === 'string') {text += block.text + '\n';}+ }+ } else if (typeof ev?.text === 'string') {+ text += ev.text + '\n';+ }+ } catch {+ /* skip malformed line */+ }+ if (text.length > cap) {break;}+ }+ return text;+}++function subcommand(argv: string[]): string | null {+ return argv.find(a => !a.startsWith('-')) ?? null;+}++function readInvocationLog(logPath: string): {commands: string[]; subs: string[]} {+ if (!fs.existsSync(logPath)) {return {commands: [], subs: []};}+ const entries = fs+ .readFileSync(logPath, 'utf-8')+ .trim()+ .split('\n')+ .filter(Boolean)+ .map(l => {+ try {+ return JSON.parse(l) as {argv: string[]};+ } catch {+ return null;+ }+ })+ .filter((e): e is {argv: string[]} => e !== null);+ return {+ commands: entries.map(e => e.argv.join(' ')),+ subs: entries.map(e => subcommand(e.argv)).filter((s): s is string => s !== null),+ };+}++interface TaskRecord {+ taskId: string;+ basePromptId: string;+ category: string;+ condition: string;+ runDir: string;+ outputFile?: string;+ invocationLog: string;+}++function evaluateRun(task: TaskRecord): RunResult {+ const versions = readSnapshots(task.runDir);+ // Fallback: if no snapshots were captured but a final file exists, classify that.+ if (versions.length === 0 && task.outputFile && fs.existsSync(task.outputFile)) {+ const finalContent = fs.readFileSync(task.outputFile, 'utf-8');+ if (finalContent.trim()) {versions.push(finalContent);}+ }+ const transcriptText = readTranscriptText(task.runDir);+ const cls = classifyTimeline(versions, {transcriptText});++ const {commands, subs} = readInvocationLog(task.invocationLog);++ let status = 'unknown';+ let model = 'unknown';+ let usage: TokenUsage | null = null;+ let estCostUSD: number | null = null;+ let apiDurationMs: number | null = null;+ const runJsonPath = path.join(task.runDir, 'run.json');+ if (fs.existsSync(runJsonPath)) {+ try {+ const rj = JSON.parse(fs.readFileSync(runJsonPath, 'utf-8'));+ status = rj.status ?? 'unknown';+ model = rj.model ?? 'unknown';+ usage = rj.usage ?? null;+ estCostUSD = rj.estCostUSD ?? null;+ apiDurationMs = rj.apiDurationMs ?? null;+ } catch {+ /* ignore */+ }+ }+ // Backfill from the transcript's result line for runs written before cost was+ // first-class (e.g. an in-flight/seed run), so token + $ data is available everywhere.+ if (!usage) {+ const res = parseResultEvent(path.join(task.runDir, 'transcript.jsonl'));+ if (res?.usage) {+ usage = res.usage;+ apiDurationMs = apiDurationMs ?? res.apiDurationMs;+ }+ }+ if (usage && estCostUSD == null) {+ estCostUSD = estimateCostUSD(usage, model === 'unknown' ? 'default' : model);+ }++ return {+ taskId: task.taskId,+ basePromptId: task.basePromptId,+ category: task.category,+ condition: task.condition,+ status,+ model,+ completed: !cls.noOutput,+ ranAstryxCli: commands.length > 0,+ ranDocRetrieval: subs.some(s => DOC_RETRIEVAL_CMDS.has(s)),+ cliCommands: commands,+ usage,+ estCostUSD,+ apiDurationMs,+ cls,+ };+}++function conditionRuns(condDir: string): RunResult[] {+ const tasksDir = path.join(condDir, 'tasks');+ if (!fs.existsSync(tasksDir)) {return [];}+ return fs+ .readdirSync(tasksDir)+ .filter(f => f.endsWith('.json'))+ .map(f => JSON.parse(fs.readFileSync(path.join(tasksDir, f), 'utf-8')))+ .map(evaluateRun);+}++function mean(nums: number[]): number {+ return nums.length ? nums.reduce((a, b) => a + b, 0) / nums.length : 0;+}++const fmtNum = (n: number) => Math.round(n).toLocaleString('en-US');++interface CostBlock {+ n: number;+ inputTokens: number;+ outputTokens: number;+ cacheReadTokens: number;+ cacheWriteTokens: number;+ avgInputTokens: number;+ avgOutputTokens: number;+ estCostUSD: number;+}++function costOf(runs: RunResult[]): CostBlock {+ let usage = emptyUsage();+ let estCostUSD = 0;+ let n = 0;+ for (const r of runs) {+ if (!r.usage) {continue;}+ usage = addUsage(usage, r.usage);+ estCostUSD += r.estCostUSD ?? 0;+ n++;+ }+ return {+ n,+ inputTokens: usage.inputTokens,+ outputTokens: usage.outputTokens,+ cacheReadTokens: usage.cacheReadTokens,+ cacheWriteTokens: usage.cacheWriteTokens,+ avgInputTokens: n ? Math.round(usage.inputTokens / n) : 0,+ avgOutputTokens: n ? Math.round(usage.outputTokens / n) : 0,+ estCostUSD,+ };+}++function costByPrompt(runs: RunResult[]): Record<string, CostBlock> {+ const groups: Record<string, RunResult[]> = {};+ for (const r of runs) {(groups[r.basePromptId] ??= []).push(r);}+ const out: Record<string, CostBlock> = {};+ for (const [pid, rs] of Object.entries(groups)) {out[pid] = costOf(rs);}+ return out;+}++function main() {+ const argv = process.argv.slice(2);+ const get = (flag: string) => {+ const i = argv.indexOf(flag);+ return i !== -1 ? argv[i + 1] : undefined;+ };+ const expId = get('--experiment');+ if (!expId) {+ console.error('Usage: purity-aggregate.ts --experiment <id> [--out <base>]');+ process.exit(1);+ }+ const expDir = path.join(sandboxBase(get('--out')), expId);+ const configPath = path.join(expDir, 'purity-config.json');+ if (!fs.existsSync(configPath)) {+ console.error(`No config at ${configPath}. Run setup-purity.mjs (and run-purity.ts) first.`);+ process.exit(1);+ }+ const config = JSON.parse(fs.readFileSync(configPath, 'utf-8'));+ const control: string = config.control ?? 'A-control';++ type CondReport = {+ n: number;+ completed: Rate;+ neverVeered: Rate;+ caughtGivenVeered: Rate;+ veeredUncaught: Rate;+ ranDocRetrieval: Rate;+ finalPurityMean: number;+ cost: CostBlock & {byPrompt: Record<string, CostBlock>};+ runs: unknown[];+ };+ const perCondition: Record<string, CondReport> = {};+ const summary: Record<string, unknown> = {+ experimentId: expId,+ generatedAt: new Date().toISOString(),+ control,+ conditions: perCondition,+ };++ console.log(`\nPrompt Purity — experiment ${expId}`);+ console.log(`Funnel over runs that produced output. Rates carry a Wilson 95% CI.\n`);+ console.log(+ `${'condition'.padEnd(14)} ${'completed'.padEnd(16)} ${'neverVeered'.padEnd(20)} ${'caught|veered'.padEnd(20)} ${'veeredUncaught'.padEnd(20)} ${'ranCLI'.padEnd(16)} purity`,+ );+ console.log('-'.repeat(126));++ for (const cond of config.conditions as string[]) {+ const condDir = config.conditionDirs[cond];+ const runs = conditionRuns(condDir);+ const done = runs.filter(r => r.completed);+ const veered = done.filter(r => r.cls.everVeered);++ const completed = rate(runs, r => r.completed);+ const neverVeered = rate(done, r => r.cls.neverVeered);+ const caughtGivenVeered = rate(veered, r => r.cls.caught);+ const veeredUncaught = rate(done, r => r.cls.veeredUncaught);+ const ranCli = rate(done, r => r.ranDocRetrieval);+ const purityMean = mean(done.map(r => r.cls.finalPurity));++ console.log(+ `${cond.padEnd(14)} ${fmtRate(completed).padEnd(16)} ${fmtRate(neverVeered).padEnd(20)} ${fmtRate(caughtGivenVeered).padEnd(20)} ${fmtRate(veeredUncaught).padEnd(20)} ${fmtRate(ranCli).padEnd(16)} ${purityMean.toFixed(0)}`,+ );++ perCondition[cond] = {+ n: runs.length,+ completed,+ neverVeered,+ caughtGivenVeered,+ veeredUncaught,+ ranDocRetrieval: ranCli,+ finalPurityMean: purityMean,+ cost: {...costOf(runs), byPrompt: costByPrompt(runs)},+ runs: runs.map(r => ({+ taskId: r.taskId,+ basePromptId: r.basePromptId,+ category: r.category,+ status: r.status,+ completed: r.completed,+ everVeered: r.cls.everVeered,+ caught: r.cls.caught,+ veeredUncaught: r.cls.veeredUncaught,+ neverVeered: r.cls.neverVeered,+ finalPurity: r.cls.finalPurity,+ peakHardCount: r.cls.peakHardCount,+ finalHardCount: r.cls.finalHardCount,+ removedHardCount: r.cls.removedHardCount,+ finalMarkers: r.cls.finalMarkers,+ ranDocRetrieval: r.ranDocRetrieval,+ cliCommands: r.cliCommands,+ caughtInReasoning: r.cls.caughtInReasoning,+ usage: r.usage,+ estCostUSD: r.estCostUSD,+ apiDurationMs: r.apiDurationMs,+ })),+ };+ }++ // ── Comparison vs control ──────────────────────────────────────────+ const base = perCondition[control];+ if (base) {+ console.log(`\nvs control (${control}) — lower veeredUncaught + higher purity is better:\n`);+ for (const cond of config.conditions as string[]) {+ if (cond === control) {continue;}+ const c = perCondition[cond];+ const vuDelta = (c.veeredUncaught.pct - base.veeredUncaught.pct) * 100;+ const purDelta = c.finalPurityMean - base.finalPurityMean;+ const decisive = c.veeredUncaught.hi < base.veeredUncaught.lo;+ console.log(+ ` ${cond.padEnd(14)} veeredUncaught ${vuDelta >= 0 ? '+' : ''}${vuDelta.toFixed(0)}pp purity ${purDelta >= 0 ? '+' : ''}${purDelta.toFixed(0)} ${decisive ? 'DECISIVE (CI clears control)' : '(CIs overlap — need more reps)'}`,+ );+ }+ }++ // ── Cost (exact tokens; $ is an estimate) ──────────────────────────+ const totals = emptyUsage();+ let totalCost = 0;+ let totalRuns = 0;+ console.log(`\nCost — exact token counts; $ is an ESTIMATE (per-model table, cache-read priced low):\n`);+ console.log(+ `${'condition'.padEnd(14)} ${'runs'.padEnd(5)} ${'input'.padEnd(12)} ${'output'.padEnd(12)} ${'cacheRead'.padEnd(14)} ${'cacheWrite'.padEnd(13)} ${'avgOut'.padEnd(8)} est$`,+ );+ console.log('-'.repeat(100));+ for (const cond of config.conditions as string[]) {+ const c = perCondition[cond].cost as CostBlock;+ console.log(+ `${cond.padEnd(14)} ${String(c.n).padEnd(5)} ${fmtNum(c.inputTokens).padEnd(12)} ${fmtNum(c.outputTokens).padEnd(12)} ${fmtNum(c.cacheReadTokens).padEnd(14)} ${fmtNum(c.cacheWriteTokens).padEnd(13)} ${fmtNum(c.avgOutputTokens).padEnd(8)} $${c.estCostUSD.toFixed(2)}`,+ );+ totals.inputTokens += c.inputTokens;+ totals.outputTokens += c.outputTokens;+ totals.cacheReadTokens += c.cacheReadTokens;+ totals.cacheWriteTokens += c.cacheWriteTokens;+ totalCost += c.estCostUSD;+ totalRuns += c.n;+ }+ console.log('-'.repeat(100));+ console.log(+ `${'TOTAL'.padEnd(14)} ${String(totalRuns).padEnd(5)} ${fmtNum(totals.inputTokens).padEnd(12)} ${fmtNum(totals.outputTokens).padEnd(12)} ${fmtNum(totals.cacheReadTokens).padEnd(14)} ${fmtNum(totals.cacheWriteTokens).padEnd(13)} ${''.padEnd(8)} $${totalCost.toFixed(2)}`,+ );+ console.log(`\n($ is an estimate — on a Cursor plan real billing may be request-based; token counts above are exact.)`);++ summary.cost = {+ total: {+ runs: totalRuns,+ inputTokens: totals.inputTokens,+ outputTokens: totals.outputTokens,+ cacheReadTokens: totals.cacheReadTokens,+ cacheWriteTokens: totals.cacheWriteTokens,+ estCostUSD: totalCost,+ },+ note: 'Token counts are exact (cursor-agent stream-json result events). estCostUSD is an estimate from the per-model PRICING table in purity-eval.ts; re-scale for graphs.',+ };++ const outPath = path.join(expDir, 'purity-summary.json');+ fs.writeFileSync(outPath, JSON.stringify(summary, null, 2));+ console.log(`\nSummary: ${outPath}`);+ console.log(`Read: a variant wins when its veeredUncaught CI clears control's and purity is higher, across K>=5 reps.\n`);+}++main();diff --git a/internal/vibe-tests/prompt-purity-test/purity-eval.ts b/internal/vibe-tests/prompt-purity-test/purity-eval.tsnew file mode 100644index 0000000000..75270c642f--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/purity-eval.ts@@ -0,0 +1,343 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file purity-eval.ts+ * @input generated .tsx code (single version) or an ordered array of versions (a run's snapshot timeline)+ * @output veer-marker hits, a 0-100 purity score, and a per-run veered/caught classification+ * @position internal/vibe-tests/prompt-purity-test — the "did it veer, did we catch it" detector+ *+ * This is the ONE place that defines what counts as "veering off Astryx". Tune here.+ *+ * HARD markers (drive the binary "veered" flag): unambiguous non-Astryx styling —+ * className=, inline style={{...}}, imported .css / @apply / @import, Tailwind+ * utility classes. In the Astryx (stylex) path these should never appear; styling+ * is Astryx components + the `xstyle` prop + tokens.+ * SOFT markers (affect the score, NOT the binary flag): hardcoded #hex/rgb() colors+ * and px/rem sizes. Kept soft because a hex inside a theme/token definition+ * (createTheme/defineVars) is legitimate Astryx, and we don't want to punish that.+ */++import * as fs from 'node:fs';++export type VeerType =+ | 'className'+ | 'inlineStyle'+ | 'cssImport'+ | 'tailwind'+ | 'hardcodedColor'+ | 'hardcodedSize';++export type Severity = 'hard' | 'soft';++export interface MarkerHit {+ type: VeerType;+ severity: Severity;+ count: number;+ example: string;+}++export interface Classification {+ versions: number;+ noOutput: boolean;+ everVeered: boolean;+ finalClean: boolean;+ /** everVeered && finalClean — tried CSS/Tailwind, then fixed it (the self-check working). */+ caught: boolean;+ /** everVeered && !finalClean — non-Astryx styling still present at the end (bad). */+ veeredUncaught: boolean;+ /** produced clean output and never introduced a hard marker (best). */+ neverVeered: boolean;+ finalPurity: number;+ peakHardCount: number;+ finalHardCount: number;+ /** how many hard veers were present at peak but gone by the end (evidence of catching). */+ removedHardCount: number;+ finalMarkers: MarkerHit[];+ /** secondary, transcript-based: agent mentioned CSS/Tailwind in reasoning. */+ mentionedVeerInReasoning: boolean;+ /** mentioned a veer but never wrote a hard marker — caught itself pre-write. */+ caughtInReasoning: boolean;+}++// ── className string extraction (for Tailwind detection) ──────────────++/** Pull the contents of className="..." / '...' / {`...`} literals. */+function extractClassNameLiterals(code: string): string[] {+ const out: string[] = [];+ const re = /className\s*=\s*(?:"([^"]*)"|'([^']*)'|\{\s*`([^`]*)`\s*\}|\{\s*"([^"]*)"\s*\}|\{\s*'([^']*)'\s*\})/g;+ let m: RegExpExecArray | null;+ while ((m = re.exec(code)) !== null) {+ const val = m[1] ?? m[2] ?? m[3] ?? m[4] ?? m[5];+ if (val) {out.push(val);}+ }+ return out;+}++const TW_TOKEN =+ /^(?:(?:sm|md|lg|xl|2xl|hover|focus|focus-visible|active|disabled|group-hover|dark|first|last|odd|even)?:)*(?:flex|grid|block|inline-block|inline-flex|inline|hidden|container|isolate|(?:p|m|px|py|pt|pb|pl|pr|mx|my|mt|mb|ml|mr|gap|gap-x|gap-y|space-x|space-y|w|h|min-w|max-w|min-h|max-h|top|bottom|left|right|inset|size)-[\w./%[\]#-]+|text-(?:xs|sm|base|lg|xl|\dxl|left|center|right|justify|[a-z]+-\d{2,3})|leading-\w+|tracking-\w+|font-(?:thin|extralight|light|normal|medium|semibold|bold|extrabold|black|mono|sans|serif)|bg-[\w./%[\]#-]+|border(?:-[\w./%[\]#-]+)?|rounded(?:-[\w]+)?|shadow(?:-[\w]+)?|ring(?:-[\w]+)?|items-\w+|justify-\w+|content-\w+|self-\w+|place-\w+|flex-\w+|grow(?:-\d)?|shrink(?:-\d)?|basis-[\w./%-]+|order-\w+|col-\w+|row-\w+|grid-cols-\w+|grid-rows-\w+|z-\d+|opacity-\d+|cursor-\w+|select-\w+|pointer-events-\w+|overflow-\w+|whitespace-\w+|truncate|uppercase|lowercase|capitalize|italic|underline|absolute|relative|fixed|sticky|static|transition(?:-\w+)?|duration-\d+|ease-\w+|animate-\w+|divide-[\w./%-]+)$/;++function countTailwindTokens(code: string): {count: number; example: string} {+ let count = 0;+ let example = '';+ for (const literal of extractClassNameLiterals(code)) {+ for (const tok of literal.split(/\s+/).filter(Boolean)) {+ if (TW_TOKEN.test(tok)) {+ count++;+ if (!example) {example = tok;}+ }+ }+ }+ return {count, example};+}++// ── Marker detectors ─────────────────────────────────────────────────++interface Detector {+ type: VeerType;+ severity: Severity;+ detect: (code: string) => {count: number; example: string};+}++function regexDetector(type: VeerType, severity: Severity, re: RegExp): Detector {+ return {+ type,+ severity,+ detect: (code: string) => {+ const matches = code.match(re) || [];+ return {count: matches.length, example: matches[0]?.trim().slice(0, 60) ?? ''};+ },+ };+}++const DETECTORS: Detector[] = [+ regexDetector('className', 'hard', /\bclassName\s*=/g),+ regexDetector('inlineStyle', 'hard', /\bstyle\s*=\s*\{\{/g),+ regexDetector('cssImport', 'hard', /(?:import[^;\n]*["'][^"']+\.css["']|@apply\b|@import\b)/g),+ {type: 'tailwind', severity: 'hard', detect: countTailwindTokens},+ regexDetector('hardcodedColor', 'soft', /#[0-9a-fA-F]{3,8}\b|\b(?:rgb|rgba|hsl|hsla)\s*\(/g),+ regexDetector('hardcodedSize', 'soft', /\b\d{2,}(?:px|rem)\b/g),+];++/** All marker hits (count > 0) for a single code string. */+export function detectMarkers(code: string): MarkerHit[] {+ const hits: MarkerHit[] = [];+ for (const d of DETECTORS) {+ const {count, example} = d.detect(code);+ if (count > 0) {hits.push({type: d.type, severity: d.severity, count, example});}+ }+ return hits;+}++export function hardMarkers(code: string): MarkerHit[] {+ return detectMarkers(code).filter(h => h.severity === 'hard');+}++export function hardMarkerCount(code: string): number {+ return hardMarkers(code).reduce((n, h) => n + h.count, 0);+}++// Per-type penalty weight and cap for the graded purity score.+const PENALTY: Record<VeerType, {weight: number; cap: number}> = {+ className: {weight: 10, cap: 40},+ inlineStyle: {weight: 10, cap: 40},+ cssImport: {weight: 20, cap: 40},+ tailwind: {weight: 3, cap: 30},+ hardcodedColor: {weight: 4, cap: 20},+ hardcodedSize: {weight: 2, cap: 16},+};++/** 0-100 purity of a single code string (100 = pure Astryx, 0 = drowning in raw CSS/TW). */+export function purityScore(code: string): number {+ if (!code || !code.trim()) {return 0;}+ let penalty = 0;+ for (const h of detectMarkers(code)) {+ const {weight, cap} = PENALTY[h.type];+ penalty += Math.min(cap, h.count * weight);+ }+ return Math.max(0, Math.min(100, 100 - penalty));+}++const REASONING_VEER =+ /\b(tailwind|plain css|raw css|vanilla css|inline styles?|hand-?roll|hardcod\w*|className)\b/i;++/**+ * Classify a run from its ordered snapshot timeline (each entry = the full file+ * contents at that moment). `transcriptText` is optional corroborating reasoning text.+ */+export function classifyTimeline(+ versions: string[],+ opts: {transcriptText?: string} = {},+): Classification {+ const nonEmpty = versions.filter(v => v && v.trim().length > 0);+ const noOutput = nonEmpty.length === 0;++ const hardCounts = versions.map(v => hardMarkerCount(v));+ const everVeered = hardCounts.some(n => n > 0);+ const peakHardCount = hardCounts.length ? Math.max(...hardCounts) : 0;++ const final = versions.length ? versions[versions.length - 1] : '';+ const finalMarkers = detectMarkers(final);+ const finalHardCount = finalMarkers.filter(h => h.severity === 'hard').reduce((n, h) => n + h.count, 0);+ const finalClean = !noOutput && finalHardCount === 0;+ const finalPurity = purityScore(final);++ const caught = everVeered && finalClean;+ const veeredUncaught = everVeered && !finalClean;+ const neverVeered = !everVeered && !noOutput;++ const mentionedVeerInReasoning = opts.transcriptText+ ? REASONING_VEER.test(opts.transcriptText)+ : false;+ const caughtInReasoning = mentionedVeerInReasoning && neverVeered;++ return {+ versions: versions.length,+ noOutput,+ everVeered,+ finalClean,+ caught,+ veeredUncaught,+ neverVeered,+ finalPurity,+ peakHardCount,+ finalHardCount,+ removedHardCount: Math.max(0, peakHardCount - finalHardCount),+ finalMarkers,+ mentionedVeerInReasoning,+ caughtInReasoning,+ };+}++/**+ * Supplementary graded quality via the shared deterministic scorer. Lazy + guarded so+ * the core classifier stays dependency-free (and the no-LLM smoke test never needs it).+ */+export async function gradedQuality(code: string): Promise<number | null> {+ try {+ const mod = (await import('../src/universal-eval.js')) as {+ evaluate: (code: string, target: string) => unknown;+ getAverageScore?: (score: unknown) => number;+ };+ const score = mod.evaluate(code, 'astryx');+ return typeof mod.getAverageScore === 'function' ? mod.getAverageScore(score) : null;+ } catch {+ return null;+ }+}++// ── Cost: parse the cursor-agent terminal `result` event + estimate $ ─────────+//+// Token counts are EXACT (from the CLI's stream-json result line). The $ is an+// ESTIMATE from the per-model table below — on a Cursor plan actual billing may be+// request-based, so treat $ as directional and re-scale PRICING for your graphs.+// The raw token counts are the hard numbers to plot.++export interface TokenUsage {+ inputTokens: number;+ outputTokens: number;+ cacheReadTokens: number;+ cacheWriteTokens: number;+}++export interface AgentResult {+ subtype: string | null;+ isError: boolean;+ durationMs: number | null;+ apiDurationMs: number | null;+ usage: TokenUsage | null;+}++export interface ModelPricing {+ inputPerM: number;+ outputPerM: number;+ cacheReadPerM: number;+ cacheWritePerM: number;+}++/** USD per 1M tokens. Estimates — edit freely; cache-read is intentionally cheap. */+export const PRICING: Record<string, ModelPricing> = {+ // Opus 4.x class+ 'claude-opus-4-8-max-fast': {inputPerM: 15, outputPerM: 75, cacheReadPerM: 1.5, cacheWritePerM: 18.75},+ 'claude-opus-4-8-thinking-max-fast': {inputPerM: 15, outputPerM: 75, cacheReadPerM: 1.5, cacheWritePerM: 18.75},+ default: {inputPerM: 15, outputPerM: 75, cacheReadPerM: 1.5, cacheWritePerM: 18.75},+};++export function pricingFor(model: string): ModelPricing {+ return PRICING[model] ?? PRICING.default;+}++export function estimateCostUSD(usage: TokenUsage, model: string): number {+ const p = pricingFor(model);+ return (+ (usage.inputTokens * p.inputPerM ++ usage.outputTokens * p.outputPerM ++ usage.cacheReadTokens * p.cacheReadPerM ++ usage.cacheWriteTokens * p.cacheWritePerM) /+ 1_000_000+ );+}++const numOrNull = (v: unknown): number | null => (typeof v === 'number' && Number.isFinite(v) ? v : null);++/**+ * Parse the last `type:"result"` line out of a cursor-agent stream-json transcript.+ * Tolerant to camelCase (cursor-agent) and snake_case (Claude-style) field names.+ */+export function parseResultFromText(text: string): AgentResult | null {+ if (!text) {return null;}+ const lines = text.split('\n');+ for (let i = lines.length - 1; i >= 0; i--) {+ const line = lines[i];+ if (!line.includes('"type":"result"') && !line.includes('"type": "result"')) {continue;}+ try {+ const ev = JSON.parse(line) as {+ type?: string;+ subtype?: string;+ is_error?: boolean;+ duration_ms?: number;+ duration_api_ms?: number;+ usage?: Record<string, unknown>;+ };+ if (ev?.type !== 'result') {continue;}+ const u: Record<string, unknown> = ev.usage ?? {};+ const usage: TokenUsage = {+ inputTokens: numOrNull(u.inputTokens ?? u.input_tokens) ?? 0,+ outputTokens: numOrNull(u.outputTokens ?? u.output_tokens) ?? 0,+ cacheReadTokens: numOrNull(u.cacheReadTokens ?? u.cache_read_input_tokens ?? u.cacheReadInputTokens) ?? 0,+ cacheWriteTokens:+ numOrNull(u.cacheWriteTokens ?? u.cache_creation_input_tokens ?? u.cacheWriteInputTokens) ?? 0,+ };+ return {+ subtype: ev.subtype ?? null,+ isError: !!ev.is_error,+ durationMs: numOrNull(ev.duration_ms),+ apiDurationMs: numOrNull(ev.duration_api_ms) ?? numOrNull(ev.duration_ms),+ usage,+ };+ } catch {+ /* malformed line — keep scanning upward */+ }+ }+ return null;+}++export function parseResultEvent(transcriptPath: string): AgentResult | null {+ try {+ return parseResultFromText(fs.readFileSync(transcriptPath, 'utf-8'));+ } catch {+ return null;+ }+}++export function emptyUsage(): TokenUsage {+ return {inputTokens: 0, outputTokens: 0, cacheReadTokens: 0, cacheWriteTokens: 0};+}++export function addUsage(a: TokenUsage, b: TokenUsage): TokenUsage {+ return {+ inputTokens: a.inputTokens + b.inputTokens,+ outputTokens: a.outputTokens + b.outputTokens,+ cacheReadTokens: a.cacheReadTokens + b.cacheReadTokens,+ cacheWriteTokens: a.cacheWriteTokens + b.cacheWriteTokens,+ };+}diff --git a/internal/vibe-tests/prompt-purity-test/run-purity.mjs b/internal/vibe-tests/prompt-purity-test/run-purity.mjsnew file mode 100644index 0000000000..2495c73aea--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/run-purity.mjs@@ -0,0 +1,78 @@+#!/usr/bin/env node+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file run-purity.mjs+ * @input CLI flags (forwarded): --conditions, --prompts, --sample, --reps, --out, --concurrency, --model, --dry-run+ * @output Runs setup -> SDK run -> aggregate for one fresh experiment id+ * @position internal/vibe-tests/prompt-purity-test — one-command orchestrator+ *+ * This is the unit the /loop command would drive: tweak variants/*.md, then re-run+ * this to get a fresh A/B comparison. Requires the cursor-agent CLI to be logged in+ * (`cursor-agent status`) unless --dry-run.+ *+ * Usage:+ * node run-purity.mjs # A vs B, pilot prompts, 3 reps+ * node run-purity.mjs --conditions A-control,B-selfcheck,C-strong --sample 6 --reps 5 --concurrency 4+ * node run-purity.mjs --dry-run # wire-check without model calls+ *+ * For AUTONOMOUS iteration, don't drive this file directly with /loop — follow the+ * playbook, which self-paces (event-gated on each run completing), evolves challenger+ * variants, confirms at K>=5, and ships the winner:+ * /loop follow internal/vibe-tests/prompt-purity-test/ITERATE.md+ */++import * as crypto from 'node:crypto';+import * as path from 'node:path';+import {execFileSync} from 'node:child_process';+import {fileURLToPath} from 'node:url';++const EXP_DIR = path.dirname(fileURLToPath(import.meta.url));++function passthrough(argv, flag) {+ const i = argv.indexOf(flag);+ return i !== -1 ? argv[i + 1] : undefined;+}++function forward(argv, flags) {+ const out = [];+ for (const f of flags) {+ const v = passthrough(argv, f);+ if (v !== undefined) out.push(f, v);+ }+ return out;+}++function run(cmd, args) {+ console.log(`\n$ ${cmd} ${args.join(' ')}\n`);+ execFileSync(cmd, args, {stdio: 'inherit', cwd: EXP_DIR});+}++function main() {+ const argv = process.argv.slice(2);+ const dryRun = argv.includes('--dry-run');+ const expId = passthrough(argv, '--exp') ?? crypto.randomBytes(4).toString('hex');++ // 1. setup — build isolated sandbox projects per condition x prompt x rep+ run('node', [+ path.join(EXP_DIR, 'setup-purity.mjs'),+ '--exp', expId,+ ...forward(argv, ['--out', '--conditions', '--prompts', '--sample', '--reps']),+ ]);++ // 2. run — Cursor SDK, captures snapshots + transcript per run+ run('npx', [+ 'tsx',+ path.join(EXP_DIR, 'run-purity.ts'),+ '--experiment', expId,+ ...forward(argv, ['--out', '--concurrency', '--model', '--conditions', '--timeout']),+ ...(dryRun ? ['--dry-run'] : []),+ ]);++ // 3. aggregate — per-condition Wilson-CI funnel + comparison vs control+ run('npx', ['tsx', path.join(EXP_DIR, 'purity-aggregate.ts'), '--experiment', expId, ...forward(argv, ['--out'])]);++ console.log(`\nExperiment ${expId} complete.`);+}++main();diff --git a/internal/vibe-tests/prompt-purity-test/run-purity.ts b/internal/vibe-tests/prompt-purity-test/run-purity.tsnew file mode 100644index 0000000000..7d3332a450--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/run-purity.ts@@ -0,0 +1,367 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file run-purity.ts+ * @input --experiment <id> [--out <base>] [--concurrency N] [--model <id>] [--timeout ms] [--conditions ...] [--dry-run]+ * @output Per task: <runDir>/transcript.jsonl, <runDir>/snapshots/NNNN-*.tsx, <runDir>/run.json+ * @position internal/vibe-tests/prompt-purity-test — the runner+ *+ * Runs each task's context-free agent via the `cursor-agent` CLI (headless) on Opus 4.8+ * 1M Max Fast, capturing the FULL trajectory so purity-aggregate.ts can tell whether the+ * agent veered off Astryx and whether it caught itself.+ *+ * Why cursor-agent (not the SDK): it ships with Cursor, uses your existing login (no npm+ * dependency, no registry/API-key setup), and `--print --output-format stream-json`+ * streams every message + tool call. Confirm auth with `cursor-agent status`.+ *+ * ISOLATION (never interferes with parallel Cursor work):+ * - Each agent runs with cwd + --workspace = the sandbox project (under $TMPDIR); it+ * can't touch your main checkout. Repo packages are only symlinked (read-only).+ * - fs.watch watches only the sandbox project dir.+ * - Bounded concurrency (--concurrency, default 3), per-run timeout, killed on stop.+ *+ * Usage:+ * npx tsx run-purity.ts --experiment <id>+ * npx tsx run-purity.ts --experiment <id> --concurrency 4 --model claude-opus-4-8-max-fast+ * npx tsx run-purity.ts --experiment <id> --dry-run # exercise the pipeline, no model calls+ */++import * as fs from 'node:fs';+import * as os from 'node:os';+import * as path from 'node:path';+import {spawn, spawnSync} from 'node:child_process';+import {parseResultFromText, estimateCostUSD, type TokenUsage, type AgentResult} from './purity-eval.js';++const DEFAULT_MODEL = 'claude-opus-4-8-max-fast'; // Opus 4.8 1M Max Fast+const DEFAULT_AGENT = 'cursor-agent';+const DEFAULT_TIMEOUT_MS = 15 * 60 * 1000;++const ensureDir = (d: string) => fs.mkdirSync(d, {recursive: true});+const writeJson = (p: string, o: unknown) => fs.writeFileSync(p, JSON.stringify(o, null, 2));++function sandboxBase(out?: string): string {+ return out || process.env.ASTRYX_PURITY_OUT || path.join(os.tmpdir(), 'astryx-purity');+}++interface TaskFile {+ taskId: string;+ basePromptId: string;+ category: string;+ condition: string;+ projectDir: string;+ outputFile: string;+ runDir: string;+ taskPrompt: string;+}++interface RunJson {+ taskId: string;+ condition: string;+ basePromptId: string;+ category: string;+ model: string;+ status: string;+ error: string | null;+ subtype: string | null;+ isError: boolean;+ startedAt: string;+ finishedAt: string;+ durationMs: number;+ apiDurationMs: number | null;+ usage: TokenUsage | null;+ estCostUSD: number | null;+ snapshotCount: number;+ outputExists: boolean;+}++// ── Live snapshotting of the output file (the purity timeline) ────────++function startSnapshotter(targetFile: string, snapDir: string) {+ ensureDir(snapDir);+ let seq = 0;+ let last: string | null = null;+ let debounce: NodeJS.Timeout | null = null;++ const capture = () => {+ try {+ if (!fs.existsSync(targetFile)) {return;}+ const content = fs.readFileSync(targetFile, 'utf-8');+ if (content === last) {return;}+ last = content;+ const name = `${String(seq++).padStart(4, '0')}-${Date.now()}.tsx`;+ fs.writeFileSync(path.join(snapDir, name), content);+ } catch {+ /* transient read during write — the poll will catch the settled version */+ }+ };++ const dir = path.dirname(targetFile);+ ensureDir(dir);+ const base = path.basename(targetFile);+ let watcher: fs.FSWatcher | null = null;+ try {+ watcher = fs.watch(dir, (_event, filename) => {+ if (filename && filename.toString() !== base) {return;}+ if (debounce) {clearTimeout(debounce);}+ debounce = setTimeout(capture, 80);+ });+ } catch {+ /* fs.watch unsupported on some FS — the poll below is the fallback */+ }+ const poll = setInterval(capture, 500);++ return {+ stop: () => {+ if (debounce) {clearTimeout(debounce);}+ clearInterval(poll);+ try {+ watcher?.close();+ } catch {+ /* ignore */+ }+ capture(); // guaranteed final snapshot+ },+ };+}++// ── Bounded concurrency pool ─────────────────────────────────────────++async function pool<T, R>(items: T[], limit: number, worker: (item: T, i: number) => Promise<R>): Promise<R[]> {+ const results = new Array<R>(items.length);+ let cursor = 0;+ const runners = Array.from({length: Math.min(Math.max(1, limit), items.length || 1)}, async () => {+ while (true) {+ const i = cursor++;+ if (i >= items.length) {break;}+ results[i] = await worker(items[i], i);+ }+ });+ await Promise.all(runners);+ return results;+}++// ── One run (spawn cursor-agent headless) ────────────────────────────++function runOne(cfg: {model: string; agent: string; timeoutMs: number}, task: TaskFile): Promise<RunJson> {+ return new Promise(resolve => {+ ensureDir(task.runDir);+ const transcriptPath = path.join(task.runDir, 'transcript.jsonl');+ const snapDir = path.join(task.runDir, 'snapshots');+ const snap = startSnapshotter(task.outputFile, snapDir);+ const startedAt = new Date().toISOString();+ const t0 = Date.now();++ let done = false;+ let transcriptBuf = '';+ const finalize = (status: string, error: string | null) => {+ if (done) {return;}+ done = true;+ snap.stop();+ const snapshotCount = fs.existsSync(snapDir) ? fs.readdirSync(snapDir).filter(f => f.endsWith('.tsx')).length : 0;+ const outputExists = fs.existsSync(task.outputFile) && fs.readFileSync(task.outputFile, 'utf-8').trim().length > 0;+ // Parse the CLI's terminal `result` line for exact token usage + api duration,+ // so cost data is first-class in run.json (durable even if transcripts are pruned).+ const result: AgentResult | null = parseResultFromText(transcriptBuf);+ const usage: TokenUsage | null = result?.usage ?? null;+ const estCostUSD = usage ? estimateCostUSD(usage, cfg.model) : null;+ const runJson: RunJson = {+ taskId: task.taskId,+ condition: task.condition,+ basePromptId: task.basePromptId,+ category: task.category,+ model: cfg.model,+ status,+ error,+ subtype: result?.subtype ?? null,+ isError: result?.isError ?? status !== 'finished',+ startedAt,+ finishedAt: new Date().toISOString(),+ durationMs: Date.now() - t0,+ apiDurationMs: result?.apiDurationMs ?? null,+ usage,+ estCostUSD,+ snapshotCount,+ outputExists,+ };+ writeJson(path.join(task.runDir, 'run.json'), runJson);+ const tok = usage ? ` tok(out=${usage.outputTokens} cacheR=${usage.cacheReadTokens})` : '';+ console.log(` [${status.padEnd(13)}] ${task.condition}/${task.taskId} snapshots=${snapshotCount} output=${outputExists}${tok}${error ? ` (${error.slice(0, 80)})` : ''}`);+ resolve(runJson);+ };++ const args = [+ '--print',+ '--output-format', 'stream-json',+ '--model', cfg.model,+ '--force', // allow write/shell non-interactively+ '--trust', // trust the sandbox workspace (headless)+ '--workspace', task.projectDir,+ task.taskPrompt,+ ];++ let child;+ try {+ child = spawn(cfg.agent, args, {cwd: task.projectDir, env: process.env});+ } catch (e) {+ finalize('startup_error', e instanceof Error ? e.message : String(e));+ return;+ }++ const out = fs.createWriteStream(transcriptPath);+ child.stdout.on('data', d => {+ const s = d.toString();+ if (transcriptBuf.length < 8_000_000) {transcriptBuf += s;} // keep in memory to parse the result line+ out.write(s); // persist full transcript to disk+ });+ let stderr = '';+ child.stderr.on('data', d => {+ if (stderr.length < 4000) {stderr += d.toString();}+ });++ let killed = false;+ const timer = setTimeout(() => {+ killed = true;+ try {+ child.kill('SIGTERM');+ } catch {+ /* ignore */+ }+ setTimeout(() => {+ try {+ child.kill('SIGKILL');+ } catch {+ /* ignore */+ }+ }, 5000);+ }, cfg.timeoutMs);++ child.on('error', (e: NodeJS.ErrnoException) => {+ clearTimeout(timer);+ finalize(e.code === 'ENOENT' ? 'startup_error' : 'exception', e.message || String(e));+ });+ child.on('close', code => {+ clearTimeout(timer);+ out.end();+ const status = killed ? 'timeout' : code === 0 ? 'finished' : 'error';+ finalize(status, status === 'finished' ? null : stderr.trim().slice(-300) || `exit ${code}`);+ });+ });+}++// ── Main ─────────────────────────────────────────────────────────────++function parseArgs(argv: string[]) {+ const get = (flag: string) => {+ const i = argv.indexOf(flag);+ return i !== -1 ? argv[i + 1] : undefined;+ };+ const getInt = (flag: string, def: number) => {+ const v = get(flag);+ return v ? parseInt(v, 10) : def;+ };+ const conditions = get('--conditions');+ return {+ experiment: get('--experiment'),+ out: get('--out'),+ concurrency: getInt('--concurrency', 3),+ model: get('--model') ?? DEFAULT_MODEL,+ agent: get('--agent') ?? DEFAULT_AGENT,+ timeoutMs: getInt('--timeout', DEFAULT_TIMEOUT_MS),+ conditions: conditions ? conditions.split(',').map(s => s.trim()) : undefined,+ dryRun: argv.includes('--dry-run'),+ };+}++function loadTasks(configDirs: Record<string, string>, conditions: string[]): TaskFile[] {+ const perCond: TaskFile[][] = [];+ for (const cond of conditions) {+ const condDir = configDirs[cond];+ if (!condDir) {throw new Error(`Condition "${cond}" not found in config.conditionDirs`);}+ const tasksDir = path.join(condDir, 'tasks');+ const arr: TaskFile[] = [];+ if (fs.existsSync(tasksDir)) {+ for (const f of fs.readdirSync(tasksDir).filter(f => f.endsWith('.json'))) {+ arr.push(JSON.parse(fs.readFileSync(path.join(tasksDir, f), 'utf-8')) as TaskFile);+ }+ }+ perCond.push(arr);+ }+ // Interleave round-robin across conditions so partial progress stays balanced and no+ // single condition monopolizes an early rate-limit window.+ const tasks: TaskFile[] = [];+ const max = Math.max(0, ...perCond.map(a => a.length));+ for (let i = 0; i < max; i++) {for (const arr of perCond) {if (i < arr.length) {tasks.push(arr[i]);}}}+ return tasks;+}++/** Confirm cursor-agent is present and logged in (non-fatal warning otherwise). */+function checkAgent(agent: string): boolean {+ const r = spawnSync(agent, ['status'], {encoding: 'utf-8'});+ if (r.error) {+ console.error(`Could not run "${agent}" (${r.error.message}). Install the Cursor CLI or pass --agent <path>.`);+ return false;+ }+ const text = `${r.stdout ?? ''}${r.stderr ?? ''}`;+ if (!/logged in|Logged in/i.test(text) && !process.env.CURSOR_API_KEY) {+ console.warn(`[warn] "${agent} status" does not report a login. Run "${agent} login" or set CURSOR_API_KEY.`);+ } else {+ console.log(` Auth: ${text.trim().split('\n')[0] ?? 'ok'}`);+ }+ return true;+}++async function main() {+ const args = parseArgs(process.argv.slice(2));+ if (!args.experiment) {+ console.error('Usage: run-purity.ts --experiment <id> [--out <base>] [--concurrency N] [--model <id>] [--dry-run]');+ process.exit(1);+ }+ const expDir = path.join(sandboxBase(args.out), args.experiment);+ const configPath = path.join(expDir, 'purity-config.json');+ if (!fs.existsSync(configPath)) {+ console.error(`No config at ${configPath}. Run setup-purity.mjs first.`);+ process.exit(1);+ }+ const config = JSON.parse(fs.readFileSync(configPath, 'utf-8'));+ const conditions = args.conditions ?? config.conditions;+ const tasks = loadTasks(config.conditionDirs, conditions);++ console.log(`\nPrompt Purity — run (experiment ${args.experiment})`);+ console.log(` Conditions: ${conditions.join(', ')}`);+ console.log(` Tasks: ${tasks.length}`);+ console.log(` Model: ${args.model}`);+ console.log(` Concurrency: ${args.concurrency}${args.dryRun ? ' (DRY RUN — no model calls)' : ''}\n`);++ if (args.dryRun) {+ for (const task of tasks) {+ ensureDir(task.runDir);+ writeJson(path.join(task.runDir, 'run.json'), {+ taskId: task.taskId,+ condition: task.condition,+ basePromptId: task.basePromptId,+ category: task.category,+ status: 'dry-run',+ snapshotCount: 0,+ outputExists: false,+ });+ console.log(` [dry-run ] ${task.condition}/${task.taskId}`);+ }+ console.log(`\nDry run complete. ${tasks.length} run dirs prepared.\n`);+ return;+ }++ if (!checkAgent(args.agent)) {process.exit(1);}++ const t0 = Date.now();+ const results = await pool(tasks, args.concurrency, task => runOne({model: args.model, agent: args.agent, timeoutMs: args.timeoutMs}, task));++ const byStatus: Record<string, number> = {};+ for (const r of results) {byStatus[r.status] = (byStatus[r.status] ?? 0) + 1;}+ console.log(`\nDone in ${((Date.now() - t0) / 1000).toFixed(0)}s. Status: ${Object.entries(byStatus).map(([k, v]) => `${k}=${v}`).join(', ')}`);+ console.log(`\nNext: npx tsx ${path.relative(process.cwd(), path.join(import.meta.dirname, 'purity-aggregate.ts'))} --experiment ${args.experiment}${args.out ? ` --out ${args.out}` : ''}\n`);+}++main().catch(e => {+ console.error(e);+ process.exit(1);+});diff --git a/internal/vibe-tests/prompt-purity-test/setup-purity.mjs b/internal/vibe-tests/prompt-purity-test/setup-purity.mjsnew file mode 100644index 0000000000..9b836f2c7d--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/setup-purity.mjs@@ -0,0 +1,378 @@+#!/usr/bin/env node+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file setup-purity.mjs+ * @input CLI flags: --exp, --out, --conditions, --prompts, --sample, --reps+ * @output One <sandbox>/<expId>/<conditionId>/ tree per condition + <sandbox>/<expId>/purity-config.json+ * @position internal/vibe-tests/prompt-purity-test — prompt-purity A/B experiment+ *+ * Builds isolated, realistic "installed @astryxdesign/core" projects whose ONLY+ * guidance is a VARIANT of the astryx-init prompt (the independent variable). Each+ * CONDITION = a fresh `astryx init` AGENTS.md + one variant self-check block spliced+ * inside the <!-- ASTRYX:START/END --> markers (see conditions.json + variants/*.md).+ * Only the injected prompt varies (Checker Protocol §2).+ *+ * ISOLATION: everything is created under an OUT-OF-WORKSPACE sandbox root+ * (os.tmpdir()/astryx-purity by default, --out to override) so it never touches your+ * open repo, editor file-watchers, other Cursor agents, or `git status`. Repo+ * packages are only symlinked (read-only).+ *+ * GROUND TRUTH: every project gets an `astryx` bin SHIM at node_modules/.bin that+ * logs argv to <project>/.astryx-invocations.log before exec-ing the real CLI, so we+ * know objectively whether the agent ran the CLI to self-verify.+ *+ * Usage:+ * node setup-purity.mjs # A vs B, curated pilot prompts, 3 reps+ * node setup-purity.mjs --conditions A-control,B-selfcheck,C-strong --sample 6 --reps 5+ * node setup-purity.mjs --exp abcd1234 --prompts dd-1,tc-6 --reps 3+ */++import * as fs from 'node:fs';+import * as os from 'node:os';+import * as path from 'node:path';+import * as crypto from 'node:crypto';+import {execFileSync} from 'node:child_process';+import {fileURLToPath} from 'node:url';++const __dirname = path.dirname(fileURLToPath(import.meta.url));+const EXP_DIR = __dirname; // internal/vibe-tests/prompt-purity-test+const VIBE_DIR = path.resolve(EXP_DIR, '..'); // internal/vibe-tests+const REPO_ROOT = path.resolve(VIBE_DIR, '../..');+const REAL_CLI_BIN = path.join(REPO_ROOT, 'packages', 'cli', 'bin', 'astryx.mjs');++// ── Helpers ──────────────────────────────────────────────────────────++const genId = () => crypto.randomBytes(4).toString('hex');+const timestamp = () => new Date().toISOString();+const ensureDir = dir => fs.mkdirSync(dir, {recursive: true});+const readJson = p => JSON.parse(fs.readFileSync(p, 'utf-8'));++/** Default sandbox base — OUTSIDE the repo. Overridable with --out or $ASTRYX_PURITY_OUT. */+export function sandboxBase(out) {+ return out || process.env.ASTRYX_PURITY_OUT || path.join(os.tmpdir(), 'astryx-purity');+}++function symlink(target, linkPath, type) {+ ensureDir(path.dirname(linkPath));+ if (fs.existsSync(linkPath)) return;+ fs.symlinkSync(target, linkPath, type);+}++/** Curated prompt IDs from prompts.json, explicit --prompts, or a stratified sample. */+function selectPrompts({sample, promptIds}) {+ const testSet = readJson(path.join(VIBE_DIR, 'test-sets', 'default.json'));+ const all = Array.isArray(testSet) ? testSet : testSet.prompts;+ const byId = new Map(all.map(p => [p.id, p]));+ const lookup = id => {+ const p = byId.get(id);+ if (!p) throw new Error(`Prompt id not found in default.json: ${id}`);+ return p;+ };++ if (promptIds && promptIds.length > 0) return promptIds.map(lookup);+ if (sample) return stratified(all, sample);++ const curated = readJson(path.join(EXP_DIR, 'prompts.json'));+ return curated.promptIds.map(({id}) => lookup(id));+}++function stratified(prompts, n) {+ const byCat = new Map();+ for (const p of prompts) {+ const arr = byCat.get(p.category) ?? [];+ arr.push(p);+ byCat.set(p.category, arr);+ }+ const cats = [...byCat.keys()];+ const out = [];+ for (const c of cats) {+ out.push(byCat.get(c)[Math.floor(Math.random() * byCat.get(c).length)]);+ if (out.length >= n) return out.slice(0, n);+ }+ let i = 0;+ while (out.length < n && i < cats.length * prompts.length) {+ const avail = byCat.get(cats[i % cats.length]).filter(p => !out.includes(p));+ if (avail.length) out.push(avail[Math.floor(Math.random() * avail.length)]);+ i++;+ }+ return out.slice(0, n);+}++// ── Base project + ground-truth shim ─────────────────────────────────++/**+ * Realistic "installed @astryxdesign/core" base. Crucially lists a StyleX *compiler*+ * plugin in devDependencies so the CLI's detectStylingSystem() returns 'stylex' and+ * the injected prompt recommends the Astryx xstyle/StyleX path (what we're testing) —+ * NOT the plain-CSS fallback. The plugin is only referenced in package.json (read by+ * detection); it is never installed/compiled here.+ */+function createBaseProject(projectDir) {+ ensureDir(projectDir);+ const pkg = {+ name: 'my-app',+ private: true,+ type: 'module',+ dependencies: {+ '@astryxdesign/core': '*',+ '@astryxdesign/theme-neutral': '*',+ '@stylexjs/stylex': '^0.10',+ react: '^19',+ 'react-dom': '^19',+ },+ devDependencies: {+ '@stylexjs/babel-plugin': '^0.10',+ },+ };+ fs.writeFileSync(path.join(projectDir, 'package.json'), JSON.stringify(pkg, null, 2));++ const nm = path.join(projectDir, 'node_modules');+ symlink(path.join(REPO_ROOT, 'packages', 'core'), path.join(nm, '@astryxdesign', 'core'), 'dir');+ symlink(path.join(REPO_ROOT, 'packages', 'themes', 'neutral'), path.join(nm, '@astryxdesign', 'theme-neutral'), 'dir');+ // CLI is symlinked (so `npx astryx` resolves) but NOT listed in package.json.+ symlink(path.join(REPO_ROOT, 'packages', 'cli'), path.join(nm, '@astryxdesign', 'cli'), 'dir');++ installLoggingShim(projectDir);+}++/**+ * Executable `astryx` shim in node_modules/.bin that appends each invocation to+ * <project>/.astryx-invocations.log, then execs the real CLI. Absolute paths baked in.+ */+function installLoggingShim(projectDir) {+ const binDir = path.join(projectDir, 'node_modules', '.bin');+ ensureDir(binDir);+ const logPath = path.join(projectDir, '.astryx-invocations.log');+ const shim = `#!/usr/bin/env node+// GENERATED shim — logs invocations for the prompt-purity experiment, then execs the real CLI.+import {appendFileSync} from 'node:fs';+import {spawnSync} from 'node:child_process';+const LOG = ${JSON.stringify(logPath)};+const REAL = ${JSON.stringify(REAL_CLI_BIN)};+const argv = process.argv.slice(2);+try { appendFileSync(LOG, JSON.stringify({ts: new Date().toISOString(), argv}) + '\\n'); } catch {}+const r = spawnSync(process.execPath, [REAL, ...argv], {stdio: 'inherit'});+process.exit(r.status ?? 0);+`;+ const shimPath = path.join(binDir, 'astryx');+ fs.writeFileSync(shimPath, shim);+ fs.chmodSync(shimPath, 0o755);+}++// ── Variant splice (the independent variable) ────────────────────────++const MARKER_END = '<!-- ASTRYX:END -->';+const LEGACY_MARKER_END = '<!-- XDS:END -->';++/**+ * Splice a variant's self-check text inside the injected ASTRYX block of AGENTS.md.+ * HTML comments are stripped; if nothing remains (the control), inject nothing.+ */+function applyVariant(projectDir, variantId) {+ const variantPath = path.join(EXP_DIR, 'variants', `${variantId}.md`);+ if (!fs.existsSync(variantPath)) {+ throw new Error(`Variant file not found: ${variantPath}`);+ }+ const raw = fs.readFileSync(variantPath, 'utf-8');+ const text = raw.replace(/<!--[\s\S]*?-->/g, '').trim();+ if (!text) return {injected: false}; // control / no-op++ const agentsPath = path.join(projectDir, 'AGENTS.md');+ if (!fs.existsSync(agentsPath)) {+ throw new Error(`AGENTS.md missing at ${agentsPath} — did \`astryx init\` run?`);+ }+ let content = fs.readFileSync(agentsPath, 'utf-8');+ const block = `\n\nPURITY SELF-CHECK\n${text}\n`;+ if (content.includes(MARKER_END)) {+ content = content.replace(MARKER_END, `${block}${MARKER_END}`);+ } else if (content.includes(LEGACY_MARKER_END)) {+ content = content.replace(LEGACY_MARKER_END, `${block}${LEGACY_MARKER_END}`);+ } else {+ content = content.trimEnd() + block;+ }+ fs.writeFileSync(agentsPath, content);+ return {injected: true};+}++/** Run the real `astryx init` (direct bin, NOT the shim) to write AGENTS.md. */+function runInit(projectDir) {+ execFileSync(+ process.execPath,+ [REAL_CLI_BIN, 'init', '--features', 'agents', '--agent-docs-path', 'AGENTS.md'],+ {cwd: projectDir, stdio: 'pipe'},+ );+}++// ── Task prompt (IDENTICAL across conditions except the AGENTS.md variant) ──++function taskPrompt(prompt, projectDir, outputFile) {+ return `First read AGENTS.md in this directory for component-library guidance, then follow it.++You are building a React + TypeScript (.tsx) UI using the Astryx design system. Your+project is at ${projectDir}.++## Task++${prompt.prompt}++## Output++Write a single, complete, self-contained .tsx file to:+ ${outputFile}++Then write a short metadata file next to it at:+ ${path.join(projectDir, 'result-meta.json')}++with this shape:+{+ "completedAt": "<ISO timestamp>",+ "docsRead": [<component/doc names you looked up>],+ "selfCheck": {+ "reReadOutput": <true|false>,+ "foundNonAstryx": <true|false>,+ "fixed": <true|false>+ }+}++Write ONLY valid TSX to the .tsx file (no markdown fences, no prose).`;+}++// ── Build one condition ──────────────────────────────────────────────++function buildCondition({expDir, conditionId, variantId, prompts, reps}) {+ const condDir = path.join(expDir, conditionId);+ ensureDir(path.join(condDir, 'tasks'));+ ensureDir(path.join(condDir, 'projects'));+ ensureDir(path.join(condDir, 'runs'));++ const taskIds = [];+ for (const prompt of prompts) {+ for (let k = 1; k <= reps; k++) {+ const taskId = reps > 1 ? `${prompt.id}__r${k}` : prompt.id;+ taskIds.push(taskId);+ const projectDir = path.join(condDir, 'projects', taskId);+ const outputFile = path.join(projectDir, 'output.tsx');++ createBaseProject(projectDir);+ runInit(projectDir);+ const {injected} = applyVariant(projectDir, variantId);++ const task = {+ taskId,+ basePromptId: prompt.id,+ category: prompt.category,+ complexity: prompt.complexity,+ condition: conditionId,+ variant: variantId,+ variantInjected: injected,+ prompt: prompt.prompt,+ expectedComponents: prompt.expectedComponents, // EVAL-ONLY; never placed in taskPrompt+ projectDir,+ outputFile,+ agentsPath: path.join(projectDir, 'AGENTS.md'),+ invocationLog: path.join(projectDir, '.astryx-invocations.log'),+ runDir: path.join(condDir, 'runs', taskId),+ taskPrompt: taskPrompt(prompt, projectDir, outputFile),+ createdAt: timestamp(),+ };+ fs.writeFileSync(path.join(condDir, 'tasks', `${taskId}.json`), JSON.stringify(task, null, 2));+ }+ }++ const manifest = {+ experimentCondition: conditionId,+ variant: variantId,+ createdAt: timestamp(),+ reps,+ prompts: prompts.map(p => ({id: p.id, category: p.category})),+ taskIds,+ };+ fs.writeFileSync(path.join(condDir, 'manifest.json'), JSON.stringify(manifest, null, 2));+ return {conditionId, condDir, taskIds};+}++// ── Main ─────────────────────────────────────────────────────────────++function parseArgs(argv) {+ const get = flag => {+ const i = argv.indexOf(flag);+ return i !== -1 ? argv[i + 1] : undefined;+ };+ const conditionsArg = get('--conditions');+ return {+ exp: get('--exp'),+ out: get('--out'),+ conditions: conditionsArg ? conditionsArg.split(',').map(s => s.trim()) : undefined,+ sample: get('--sample') ? parseInt(get('--sample'), 10) : undefined,+ reps: get('--reps') ? parseInt(get('--reps'), 10) : undefined,+ promptIds: get('--prompts') ? get('--prompts').split(',').map(s => s.trim()) : undefined,+ };+}++function main() {+ const args = parseArgs(process.argv.slice(2));+ const conditionsDef = readJson(path.join(EXP_DIR, 'conditions.json'));+ const byId = new Map(conditionsDef.conditions.map(c => [c.id, c]));++ const conditionIds = args.conditions ?? ['A-control', 'B-selfcheck'];+ for (const id of conditionIds) {+ if (!byId.has(id)) {+ throw new Error(`Unknown condition "${id}". Known: ${[...byId.keys()].join(', ')}`);+ }+ }++ // Pilot default prompts come from prompts.json#pilotDefault when nothing specified.+ let promptIds = args.promptIds;+ if (!promptIds && !args.sample) {+ const curated = readJson(path.join(EXP_DIR, 'prompts.json'));+ promptIds = curated.pilotDefault ?? curated.promptIds.map(p => p.id);+ }+ const reps = args.reps ?? 3;+ const prompts = selectPrompts({sample: args.sample, promptIds});++ const expId = args.exp ?? genId();+ const base = sandboxBase(args.out);+ const expDir = path.join(base, expId);+ ensureDir(expDir);++ console.log(`\nPrompt Purity Vibe Test — experiment ${expId}`);+ console.log(` Sandbox: ${expDir}`);+ console.log(` Conditions: ${conditionIds.join(', ')}`);+ console.log(` Prompts: ${prompts.map(p => p.id).join(', ')} x ${reps} rep(s)`);+ console.log(` Runs/cond: ${prompts.length * reps}\n`);++ const built = [];+ for (const id of conditionIds) {+ const c = byId.get(id);+ process.stdout.write(` building ${id} (variant ${c.variant})...`);+ built.push(buildCondition({expDir, conditionId: id, variantId: c.variant, prompts, reps}));+ process.stdout.write(' done\n');+ }++ const config = {+ experimentId: expId,+ experiment: 'prompt-purity',+ createdAt: timestamp(),+ sandboxRoot: expDir,+ control: conditionsDef.control ?? 'A-control',+ reps,+ promptIds: prompts.map(p => p.id),+ conditions: conditionIds,+ conditionDirs: Object.fromEntries(built.map(b => [b.conditionId, b.condDir])),+ };+ const configPath = path.join(expDir, 'purity-config.json');+ fs.writeFileSync(configPath, JSON.stringify(config, null, 2));++ console.log(`\nConditions ready:`);+ for (const b of built) console.log(` ${b.conditionId.padEnd(14)} ${b.taskIds.length} tasks`);+ console.log(`\nConfig: ${configPath}`);+ console.log(`\nNext:`);+ console.log(` npx tsx ${path.relative(process.cwd(), path.join(EXP_DIR, 'run-purity.ts'))} --experiment ${expId}${args.out ? ` --out ${args.out}` : ''}`);+ console.log(` npx tsx ${path.relative(process.cwd(), path.join(EXP_DIR, 'purity-aggregate.ts'))} --experiment ${expId}${args.out ? ` --out ${args.out}` : ''}\n`);++ // Machine-readable line for the orchestrator to capture the id.+ console.log(`PURITY_EXPERIMENT_ID=${expId}`);+}++main();diff --git a/internal/vibe-tests/prompt-purity-test/smoke-test.ts b/internal/vibe-tests/prompt-purity-test/smoke-test.tsnew file mode 100644index 0000000000..4568400f84--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/smoke-test.ts@@ -0,0 +1,220 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file smoke-test.ts+ * @input none (self-contained; runs setup into a temp dir)+ * @output PASS/FAIL per check; exit 1 on any failure+ * @position internal/vibe-tests/prompt-purity-test — no-LLM verification of the harness+ *+ * Verifies the harness mechanism WITHOUT any model calls:+ * A. the veer/catch classifier labels fabricated snapshot timelines correctly+ * B. setup-purity.mjs wires a real condition correctly (variant spliced into AGENTS.md,+ * stylex styling detected, logging shim installed, NO expectedComponents leak)+ * C. run-purity.ts --dry-run drives the pipeline headlessly (best-effort)+ *+ * Usage: npx tsx smoke-test.ts+ */++import * as assert from 'node:assert';+import * as fs from 'node:fs';+import * as os from 'node:os';+import * as path from 'node:path';+import * as crypto from 'node:crypto';+import {execFileSync} from 'node:child_process';+import {classifyTimeline, detectMarkers, purityScore} from './purity-eval.js';++const EXP_DIR = import.meta.dirname;+let failures = 0;+let passed = 0;++function check(name: string, fn: () => void) {+ try {+ fn();+ passed++;+ console.log(` PASS ${name}`);+ } catch (e) {+ failures++;+ console.error(` FAIL ${name}\n ${e instanceof Error ? e.message : String(e)}`);+ }+}++/** Best-effort check: reports a WARN instead of failing the suite. */+function softCheck(name: string, fn: () => void) {+ try {+ fn();+ passed++;+ console.log(` PASS ${name}`);+ } catch (e) {+ console.warn(` WARN ${name}\n ${e instanceof Error ? e.message : String(e)}`);+ }+}++// ── Fixtures ─────────────────────────────────────────────────────────++const PURE = `import {Card, VStack, Heading, Text, Button} from '@astryxdesign/core';+export default function App() {+ return (+ <Card>+ <VStack gap="md">+ <Heading level={2}>Users</Heading>+ <Text>All good.</Text>+ <Button variant="primary">Add</Button>+ </VStack>+ </Card>+ );+}`;++const DIRTY = `import './styles.css';+export default function App() {+ return (+ <div className="flex flex-col p-4 gap-2 bg-gray-100 rounded-lg" style={{color: '#ffffff', padding: '16px'}}>+ <span className="text-sm font-semibold">Users</span>+ </div>+ );+}`;++// ── A. classifier ────────────────────────────────────────────────────++console.log('\nA. Veer/catch classifier (fabricated timelines)');++check('detectMarkers finds className/inlineStyle/cssImport/tailwind/hardcodedColor in dirty code', () => {+ const types = new Set(detectMarkers(DIRTY).map(h => h.type));+ for (const t of ['className', 'inlineStyle', 'cssImport', 'tailwind', 'hardcodedColor'] as const) {+ assert.ok(types.has(t), `expected marker "${t}"`);+ }+});++check('detectMarkers finds nothing hard in pure Astryx code', () => {+ const hard = detectMarkers(PURE).filter(h => h.severity === 'hard');+ assert.equal(hard.length, 0, `unexpected hard markers: ${hard.map(h => h.type).join(', ')}`);+});++check('purityScore: pure high (>90), dirty low (<50)', () => {+ assert.ok(purityScore(PURE) > 90, `pure=${purityScore(PURE)}`);+ assert.ok(purityScore(DIRTY) < 50, `dirty=${purityScore(DIRTY)}`);+});++check('neverVeered: single clean write', () => {+ const c = classifyTimeline([PURE]);+ assert.ok(c.neverVeered && !c.everVeered && !c.caught && !c.veeredUncaught && !c.noOutput, JSON.stringify(c));+});++check('caught: dirty then clean (veered, then fixed)', () => {+ const c = classifyTimeline([DIRTY, PURE]);+ assert.ok(c.everVeered, 'should have veered');+ assert.ok(c.caught, 'should be caught');+ assert.ok(!c.veeredUncaught && !c.neverVeered, JSON.stringify(c));+ assert.ok(c.removedHardCount > 0, 'should record removed hard markers');+});++check('veeredUncaught: clean then dirty (veered, not fixed)', () => {+ const c = classifyTimeline([PURE, DIRTY]);+ assert.ok(c.everVeered && c.veeredUncaught, JSON.stringify(c));+ assert.ok(!c.caught && !c.neverVeered, JSON.stringify(c));+});++check('noOutput: empty timeline', () => {+ const c = classifyTimeline([]);+ assert.ok(c.noOutput && !c.everVeered && !c.neverVeered, JSON.stringify(c));+});++check('caughtInReasoning: mentioned Tailwind but never wrote a hard marker', () => {+ const c = classifyTimeline([PURE], {transcriptText: 'I considered using Tailwind here but Astryx has a component for it.'});+ assert.ok(c.mentionedVeerInReasoning && c.neverVeered && c.caughtInReasoning, JSON.stringify(c));+});++// ── B. setup wiring ──────────────────────────────────────────────────++console.log('\nB. setup-purity.mjs wiring (real astryx init into a temp sandbox)');++const smokeOut = fs.mkdtempSync(path.join(os.tmpdir(), 'purity-smoke-'));+const expId = 'smoke' + crypto.randomBytes(2).toString('hex');+let setupOk = false;++check('setup-purity.mjs builds A-control + B-selfcheck for dd-1 (1 rep)', () => {+ execFileSync(+ process.execPath,+ [+ path.join(EXP_DIR, 'setup-purity.mjs'),+ '--exp', expId,+ '--out', smokeOut,+ '--conditions', 'A-control,B-selfcheck',+ '--prompts', 'dd-1',+ '--reps', '1',+ ],+ {stdio: 'pipe'},+ );+ assert.ok(fs.existsSync(path.join(smokeOut, expId, 'purity-config.json')), 'purity-config.json missing');+ setupOk = true;+});++const proj = (cond: string) => path.join(smokeOut, expId, cond, 'projects', 'dd-1');++check('control AGENTS.md has NO self-check; candidate AGENTS.md HAS it', () => {+ assert.ok(setupOk, 'setup did not complete');+ const a = fs.readFileSync(path.join(proj('A-control'), 'AGENTS.md'), 'utf-8');+ const b = fs.readFileSync(path.join(proj('B-selfcheck'), 'AGENTS.md'), 'utf-8');+ assert.ok(!a.includes('PURITY SELF-CHECK'), 'control should not contain the self-check block');+ assert.ok(b.includes('PURITY SELF-CHECK'), 'candidate should contain the self-check block');+ assert.ok(b.includes('re-read the file'), 'candidate should contain the B-selfcheck text');+});++check('injected prompt recommends the Astryx xstyle/StyleX path (stylex detected)', () => {+ assert.ok(setupOk, 'setup did not complete');+ const a = fs.readFileSync(path.join(proj('A-control'), 'AGENTS.md'), 'utf-8');+ assert.ok(/xstyle|StyleX/i.test(a), 'AGENTS.md should reference xstyle/StyleX (styling system should detect as stylex)');+});++check('base package.json lists a StyleX compiler plugin (drives stylex detection)', () => {+ assert.ok(setupOk, 'setup did not complete');+ const pkg = JSON.parse(fs.readFileSync(path.join(proj('A-control'), 'package.json'), 'utf-8'));+ assert.ok(pkg.devDependencies?.['@stylexjs/babel-plugin'], 'missing @stylexjs/babel-plugin');+});++check('logging astryx shim is installed', () => {+ assert.ok(setupOk, 'setup did not complete');+ const shim = path.join(proj('A-control'), 'node_modules', '.bin', 'astryx');+ assert.ok(fs.existsSync(shim), 'shim missing');+ assert.ok(fs.readFileSync(shim, 'utf-8').includes('.astryx-invocations.log'), 'shim does not log invocations');+});++check('no answer leak: expectedComponents never appears in the task prompt', () => {+ assert.ok(setupOk, 'setup did not complete');+ const task = JSON.parse(fs.readFileSync(path.join(smokeOut, expId, 'B-selfcheck', 'tasks', 'dd-1.json'), 'utf-8'));+ assert.ok(!task.taskPrompt.includes('expectedComponents'), 'taskPrompt leaks the expectedComponents key');+ for (const comp of task.expectedComponents ?? []) {+ assert.ok(!task.taskPrompt.includes(comp), `taskPrompt leaks expected component "${comp}"`);+ }+});++// ── C. runner dry-run (best-effort) ──────────────────────────────────++console.log('\nC. run-purity.ts --dry-run (headless pipeline; best-effort)');++softCheck('dry-run creates run.json with status "dry-run"', () => {+ assert.ok(setupOk, 'setup did not complete');+ try {+ execFileSync('npx', ['tsx', path.join(EXP_DIR, 'run-purity.ts'), '--experiment', expId, '--out', smokeOut, '--dry-run'], {+ stdio: 'pipe',+ cwd: EXP_DIR,+ });+ } catch (e) {+ // Non-fatal: environment may not resolve `npx tsx` here. The classifier + wiring+ // checks are the load-bearing ones; surface this as a soft note instead.+ throw new Error(`could not run dry-run via "npx tsx" (${e instanceof Error ? e.message : String(e)}) — run it manually to verify`, {cause: e});+ }+ const runJson = path.join(smokeOut, expId, 'A-control', 'runs', 'dd-1', 'run.json');+ assert.ok(fs.existsSync(runJson), 'run.json not created by dry-run');+ assert.equal(JSON.parse(fs.readFileSync(runJson, 'utf-8')).status, 'dry-run');+});++// ── Cleanup + report ─────────────────────────────────────────────────++try {+ fs.rmSync(smokeOut, {recursive: true, force: true});+} catch {+ /* ignore */+}++console.log(`\n${failures === 0 ? 'ALL CHECKS PASSED' : `${failures} CHECK(S) FAILED`} (${passed} passed, ${failures} failed)\n`);+process.exit(failures === 0 ? 0 : 1);diff --git a/internal/vibe-tests/prompt-purity-test/variants/A-control.md b/internal/vibe-tests/prompt-purity-test/variants/A-control.mdnew file mode 100644index 0000000000..89a8e4da1f--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/variants/A-control.md@@ -0,0 +1,7 @@+<!--+Control condition: NO self-check block is appended.+This file is intentionally comment-only. setup-purity.mjs strips HTML comments and,+finding no remaining text, injects nothing — so the agent sees the astryx-init prompt+exactly as `generateCompressedIndex()` emits it today. This is the baseline the+candidate variants (B, C) are measured against.+-->diff --git a/internal/vibe-tests/prompt-purity-test/variants/B-selfcheck.md b/internal/vibe-tests/prompt-purity-test/variants/B-selfcheck.mdnew file mode 100644index 0000000000..5d40221435--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/variants/B-selfcheck.md@@ -0,0 +1,7 @@+SELF-CHECK (before you finish, every time): re-read the file you wrote. If you see+className=, style={{...}}, an imported .css / @apply, Tailwind utility classes, raw+<div>/<span> used for layout, or hardcoded #hex/rgb()/px/rem — you veered off Astryx.+For each, replace it with the Astryx component or the xstyle prop + token. If unsure a+component exists, run `astryx search "<thing>"` / `astryx component <Name>` — do not+hand-roll it in CSS. Finish only when the file is Astryx components + xstyle/tokens+with zero raw CSS/Tailwind.diff --git a/internal/vibe-tests/prompt-purity-test/variants/C-strong.md b/internal/vibe-tests/prompt-purity-test/variants/C-strong.mdnew file mode 100644index 0000000000..426ffa7c48--- /dev/null+++ b/internal/vibe-tests/prompt-purity-test/variants/C-strong.md@@ -0,0 +1,15 @@+STOP — MANDATORY SELF-AUDIT before you output the final file. Astryx is the ONLY+styling system here; raw CSS/Tailwind is a failure, not a fallback. Run this checklist+against your file and fix every hit before finishing:++1. Grep your own output for: `className=`, `style={{`, `import "*.css"`, `@apply`,+ Tailwind utilities (e.g. `flex`, `p-4`, `text-sm`, `bg-*`, `rounded-*`), raw+ `<div>/<span>` doing layout, and hardcoded values (`#hex`, `rgb(`, `NNpx`, `NNrem`).+2. For EACH hit: delete it and use the Astryx component (layout, spacing, surfaces,+ typography all have one) or the `xstyle` prop with a token. Never a raw element ++ CSS when a component exists.+3. If you are not 100% sure a component/prop exists, you MUST verify with+ `astryx component <Name>` or `astryx search "<thing>"` — do NOT guess and do NOT+ fall back to CSS.+4. Only finish when a re-read shows ZERO items from step 1. If any remain, you have+ not finished — fix them and re-audit.diff --git a/packages/cli/src/commands/agent-docs.mjs b/packages/cli/src/commands/agent-docs.mjsindex 3deef40a64..b1fb93479c 100644--- a/packages/cli/src/commands/agent-docs.mjs+++ b/packages/cli/src/commands/agent-docs.mjs@@ -197,6 +197,20 @@ export function generateCompressedIndex(version, {coreDir, runPrefix = getRunPrelines.push("- Custom styling: component props first; else style/className with tokens — var(--color-*|--spacing-*|--radius-*). No raw hex/px. (No StyleX/Tailwind compiler here — don't use xstyle/utility classes.)");}lines.push('- Tokens for every value (`astryx docs tokens`). Brand/accent via `astryx theme` — never override --color-* in :root.');+ // Self-check — post-generation pass. Validated via vibe tests (internal/vibe-tests/+ // prompt-purity-test): on complex multi-step UIs the rules above alone still leave raw+ // CSS in ~11-13% of runs; a re-read-and-fix pass cuts that ~4x at negligible token cost.+ // The fix names the sanctioned escape hatch for the configured system.+ const selfCheckFix = {+ stylex:+ 'replace any className=, style={{…}}, raw <div>/<span> layout, imported .css/@apply, or hardcoded #hex/px with the component or the xstyle prop + a token',+ tailwind:+ 'replace any style={{…}}, raw <div>/<span> layout, imported .css/@apply, or hardcoded/arbitrary value (e.g. bg-[#fff], p-[13px]) with the component or a token-backed utility',+ css: 'replace any raw <div>/<span> layout, imported .css/@apply, or hardcoded value (#hex, 16px) with the component or a token (var(--color-*|--spacing-*|…))',+ };+ lines.push(+ `- SELF-CHECK before you finish: re-read the file and ${selfCheckFix[stylingSystem] ?? selfCheckFix.css}. If unsure a component/prop exists, run \`astryx component <Name>\` / \`astryx search "<thing>"\`; don't hand-roll CSS.`,+ );lines.push('');// Command reference — build/template/component are covered in WORKFLOW above.diff --git a/packages/cli/src/commands/agent-docs.test.mjs b/packages/cli/src/commands/agent-docs.test.mjsindex bdd2c9124b..56e79317e8 100644--- a/packages/cli/src/commands/agent-docs.test.mjs+++ b/packages/cli/src/commands/agent-docs.test.mjs@@ -43,6 +43,26 @@ describe('generateCompressedIndex', () => {expect(result).toMatch(/never override --color-/);});+ it('includes the post-generation self-check rule', () => {+ const result = generateCompressedIndex('1.0.0');+ expect(result).toContain('SELF-CHECK before you finish');+ expect(result).toMatch(/re-read the file/);+ expect(result).toMatch(/don't hand-roll CSS/);+ });++ it('tailors the self-check to the styling system (xstyle for StyleX, not className for Tailwind)', () => {+ // StyleX path: className/inline style are veers; the fix is the xstyle prop + a token+ const stylex = generateCompressedIndex('1.0.0', {stylingSystem: 'stylex'});+ const stylexSelfCheck = stylex.split('\n').find(l => l.includes('SELF-CHECK'));+ expect(stylexSelfCheck).toMatch(/xstyle/);+ expect(stylexSelfCheck).toMatch(/className=/);+ // className IS the system in Tailwind — it must NOT be flagged+ const tailwind = generateCompressedIndex('1.0.0', {stylingSystem: 'tailwind'});+ const tailwindSelfCheck = tailwind.split('\n').find(l => l.includes('SELF-CHECK'));+ expect(tailwindSelfCheck).toBeDefined();+ expect(tailwindSelfCheck).not.toMatch(/className=/);+ });+it('defaults to the CSS-variable styling path (no compiler)', () => {const result = generateCompressedIndex('1.0.0');expect(result).toMatch(/style\/className with tokens/);