Skip to content

An open source design system that's fully customizable and agent ready

TypeScript

chore(vibe-tests): remove orphaned /vibe-test command doc (#3165)

Cindy Zhang411b489c1d4b
diff --git a/.claude/commands/vibe-test.md b/.claude/commands/vibe-test.md
deleted file mode 100644
index edc4b3aba3..0000000000
--- a/.claude/commands/vibe-test.md
+++ /dev/null
@@ -1,193 +0,0 @@
-# Run Vibe Tests
-
-Run vibeability tests to measure how well the XDS skill doc helps LLMs generate correct component code.
-
-## Sample Count
-
-$ARGUMENTS
-
-If no argument provided, default to 5 samples.
-
-## Instructions
-
-Run interactive vibe tests using parallel subagents. No API keys required — code generation is done by spawned sub-agents, screenshots by GitHub Actions.
-
-### Step 0: Generate skill doc
-
-Generate the skill doc from the CLI (auto-discovers components from source):
-
-```bash
-cd internal/vibe-tests && bash scripts/generate-skill-doc.sh
-```
-
-This creates `.generated/xds-skill.md` reflecting all components on the current branch.
-
-### Step 1: Setup the test iteration
-
-Three targets are compared: *XDS*, *baseline* (shadcn/Tailwind), and *raw HTML/CSS*. All three use the **same prompts** — this is the most important fairness constraint.
-
-```bash
-cd internal/vibe-tests
-
-# 1. Sample prompts for XDS (records the prompt IDs)
-pnpm interactive --target xds --persona naive --sample <count>
-# Note the iteration ID and prompt IDs from the output
-
-# 2. Reuse same prompts for baseline
-pnpm interactive --target baseline --persona naive --prompts <comma-separated-ids>
-
-# 3. Reuse same prompts for HTML
-pnpm interactive --target html --persona naive --prompts <comma-separated-ids>
-```
-
-This creates:
-- Three iteration IDs (one per target)
-- Task files in `results/<iteration>/tasks/<promptId>.json`
-- A manifest file per iteration
-
-### Step 2: Spawn sub-agents
-
-Spawn sub-agents using `spawn_agent` — one per prompt per target (e.g. 10 prompts × 3 targets = 30 agents).
-
-Each sub-agent must:
-
-1. **Read the task file** at `results/<iteration>/tasks/<promptId>.json` (on the `xds` node)
-2. **Read the target-specific agent doc:**
- - XDS → `AGENTS.md` (generated) + run `npx astryx component --brief-all` for component catalog
- - Baseline → `AGENTS.baseline.md` + read `.baseline-docs/`
- - HTML → `AGENTS.html.md` (no design system docs needed)
-3. **Generate a React component** as a `.tsx` file responding to the prompt
-4. **Self-evaluate** for correctness, accessibility, escape hatches
-5. **Write two files** (on the `xds` node):
- - `results/<iteration>/results/<promptId>.tsx` — the generated component (must have a default export)
- - `results/<iteration>/results/<promptId>.json` — metadata:
- ```json
- {
- "docsRead": ["AGENTS.md", "Button.md", "tokens.md"],
- "completedAt": "2026-03-25T06:00:00Z"
- }
- ```
-
-### Step 3: Verify completeness
-
-After all sub-agents complete, verify every task has a `.tsx` and `.json` result:
-
-```bash
-cd internal/vibe-tests
-for ITER in <xds-id> <baseline-id> <html-id>; do
- EXPECTED=$(ls results/$ITER/tasks/*.json | wc -l)
- ACTUAL=$(ls results/$ITER/results/*.tsx 2>/dev/null | wc -l)
- echo "$ITER: $ACTUAL/$EXPECTED complete"
-done
-```
-
-Log any gaps. Missing results mean a sub-agent failed — note which prompt/target is affected.
-
-### Step 4: Build previews & screenshots (GitHub Actions)
-
-Commit results to a branch and trigger the screenshot workflow:
-
-```bash
-cd /vercel/sandbox/repos/xds
-git checkout -b vibe-test-data/<date>
-git add -f internal/vibe-tests/results/
-git commit -m "vibe-test: results for $(date +%Y-%m-%d)"
-git push -u origin vibe-test-data/<date>
-
-# Trigger the screenshot workflow
-gh workflow run vibe-screenshots.yml \
- --ref vibe-test-data/<date> \
- -f iterations="<xds-id>,<baseline-id>,<html-id>"
-```
-
-The workflow:
-1. Builds self-contained HTML previews (inlines all CSS via vite-plugin-singlefile)
-2. Captures Playwright screenshots at 4 combos: desktop/mobile × light/dark
-3. Uploads artifacts: `preview-html` and `vibe-test-screenshots`
-
-### Step 5: Wait for GHA and download
-
-Poll the run until complete:
-
-```bash
-RUN_ID=$(gh run list --workflow=vibe-screenshots.yml --limit 1 --json databaseId --jq '.[0].databaseId')
-gh run watch $RUN_ID
-```
-
-Download artifacts:
-
-```bash
-gh run download $RUN_ID -n vibe-test-screenshots -D internal/vibe-tests/results/
-gh run download $RUN_ID -n preview-html -D internal/vibe-tests/results/
-```
-
-If GHA fails: `gh run view $RUN_ID --log-failed`
-
-### Step 6: Aggregate and report
-
-```bash
-cd internal/vibe-tests
-pnpm aggregate --iteration <xds-id>
-pnpm report:deploy --iteration <xds-id> --baseline <baseline-id> --html <html-id>
-```
-
-Report URL: `https://facebook.github.io/astryx/reports/<xds-id>/`
-
-### Step 7: Post results
-
-Create a GitHub issue:
-
-```bash
-gh issue create --repo facebookexperimental/xds \
- --title "Vibe Test Results — $(date +%Y-%m-%d)" \
- --label "vibe-test" \
- --body "<results summary>"
-```
-
-Include: overall scores, dimension winners, report link, gaps, and screenshots.
-
-## Fairness Rules
-
-These rules ensure the comparison is meaningful:
-
-1. **Same prompts** — all targets run the exact same prompt IDs (use `--prompts`)
-2. **Same persona** — always `naive` (simulates a developer with no prior knowledge)
-3. **Fair docs** — each target gets its own equivalent agent doc, not the other target's
-4. **No cross-contamination** — XDS agents don't see baseline docs, and vice versa
-5. **10 prompts minimum** — stratified sample covers all categories
-
-## Evaluation Criteria
-
-### Quality Tiers
-
-- 🥇 **Gold** — Pure target usage, no escape hatches needed
-- 🟢 **Green** — Correct components, only acceptable escape hatches
-- 🟡 **Yellow** — Anti-patterns that break theming (hardcoded values)
-- 🔴 **Red** — Critical failures (hallucinations, wrong components)
-
-### Escape Hatch Classification
-
-**Critical (Red tier):**
-- `hallucination` — invented props/components that don't exist
-- `wrong_component` — using a component incorrectly
-- `redundant_css` — CSS that duplicates a component prop
-
-**Anti-pattern (Yellow tier):**
-- `hardcoded_color` — direct color values instead of CSS variables
-- `hardcoded_spacing` — direct pixel values instead of CSS variables
-- `inline_style` — inline styles instead of StyleX
-
-**Acceptable (Green tier):**
-- `supplemental_css` — StyleX for things the system doesn't cover
-- `wrapper_div` — structural HTML for composition
-- `custom_animation` — animation not covered by the system
-
-## Evaluation Dimensions
-
-| Dimension | What It Measures |
-|-----------|-----------------|
-| Correctness | Valid component usage, no hallucinations |
-| Accessibility | Labels, semantics, keyboard support |
-| Code Quality | Complexity, patterns, TypeScript usage |
-| Efficiency | Decisions/element, DRYness, conciseness |
-| Maintainability | Semantic tokens vs magic values, dark mode, locality |