An open source design system that's fully customizable and agent ready
TypeScript
feat(vibe-tests): add GHA screenshot service for nightly pipeline (#853)
Cindy Zhanga1d452a2ac49
diff --git a/.claude/commands/vibe-test.md b/.claude/commands/vibe-test.mdindex 6e7a4116df..e6c108e134 100644--- a/.claude/commands/vibe-test.md+++ b/.claude/commands/vibe-test.md@@ -10,7 +10,7 @@ If no argument provided, default to 5 samples.## Instructions-Run interactive vibe tests using parallel subagents:+Run interactive vibe tests using parallel subagents. No API keys required — code generation is done by spawned sub-agents, screenshots by GitHub Actions.### Step 0: Generate skill doc@@ -24,122 +24,170 @@ This creates `.generated/xds-skill.md` reflecting all components on the current### Step 1: Setup the test iteration-Run the interactive setup command:+Three targets are compared: *XDS*, *baseline* (shadcn/Tailwind), and *raw HTML/CSS*. All three use the **same prompts** — this is the most important fairness constraint.```bash-yarn workspace @xds/vibe-tests interactive --sample <count>+cd internal/vibe-tests++# 1. Sample prompts for XDS (records the prompt IDs)+yarn interactive --target xds --persona naive --sample <count>+# Note the iteration ID and prompt IDs from the output++# 2. Reuse same prompts for baseline+yarn interactive --target baseline --persona naive --prompts <comma-separated-ids>++# 3. Reuse same prompts for HTML+yarn interactive --target html --persona naive --prompts <comma-separated-ids>```This creates:+- Three iteration IDs (one per target)+- Task files in `results/<iteration>/tasks/<promptId>.json`+- A manifest file per iteration-- A unique iteration ID-- Task files in `internal/vibe-tests/results/<iteration>/tasks/`-- A manifest file+### Step 2: Spawn sub-agents-### Step 2: Read the test prompts+Spawn sub-agents using `spawn_agent` — one per prompt per target (e.g. 10 prompts × 3 targets = 30 agents).-Read the manifest to get the list of test prompts:+Each sub-agent must:+1. **Read the task file** at `results/<iteration>/tasks/<promptId>.json` (on the `xds` node)+2. **Read the target-specific agent doc:**+ - XDS → `AGENTS.md` (generated) + run `npx xds component --brief-all` for component catalog+ - Baseline → `AGENTS.baseline.md` + read `.baseline-docs/`+ - HTML → `AGENTS.html.md` (no design system docs needed)+3. **Generate a React component** as a `.tsx` file responding to the prompt+4. **Self-evaluate** for correctness, accessibility, escape hatches+5. **Write two files** (on the `xds` node):+ - `results/<iteration>/results/<promptId>.tsx` — the generated component (must have a default export)+ - `results/<iteration>/results/<promptId>.json` — metadata:+ ```json+ {+ "docsRead": ["AGENTS.md", "Button.md", "tokens.md"],+ "completedAt": "2026-03-25T06:00:00Z"+ }+ ```++### Step 3: Verify completeness++After all sub-agents complete, verify every task has a `.tsx` and `.json` result:++```bash+cd internal/vibe-tests+for ITER in <xds-id> <baseline-id> <html-id>; do+ EXPECTED=$(ls results/$ITER/tasks/*.json | wc -l)+ ACTUAL=$(ls results/$ITER/results/*.tsx 2>/dev/null | wc -l)+ echo "$ITER: $ACTUAL/$EXPECTED complete"+done```-internal/vibe-tests/results/<iteration>/manifest.json-```-### Step 3: Run tests in parallel+Log any gaps. Missing results mean a sub-agent failed — note which prompt/target is affected.++### Step 4: Build previews & screenshots (GitHub Actions)++Commit results to a branch and trigger the screenshot workflow:-For each test prompt, spawn a subagent using the Task tool with:+```bash+cd /vercel/sandbox/repos/xds+git checkout -b vibe-test-data/<date>+git add -f internal/vibe-tests/results/+git commit -m "vibe-test: results for $(date +%Y-%m-%d)"+git push -u origin vibe-test-data/<date>++# Trigger the screenshot workflow+gh workflow run vibe-screenshots.yml \+ --ref vibe-test-data/<date> \+ -f iterations="<xds-id>,<baseline-id>,<html-id>"+```-- `subagent_type`: "general-purpose"-- Read the generated skill doc at `internal/vibe-tests/.generated/xds-skill.md`-- Generate code for the prompt using XDS components-- Self-evaluate the response for:- - Correct components used- - No hallucinated props- - Acceptable vs critical escape hatches-- Track timing: record start time before generating, calculate durationMs after-- Return JSON with: response, evaluation, durationMs, inputTokens, outputTokens+The workflow:+1. Builds self-contained HTML previews (inlines all CSS via vite-plugin-singlefile)+2. Captures Playwright screenshots at 4 combos: desktop/mobile × light/dark+3. Uploads artifacts: `preview-html` and `vibe-test-screenshots`-### Step 4: Write results+### Step 5: Wait for GHA and download-After all subagents complete, write results to:+Poll the run until complete:+```bash+RUN_ID=$(gh run list --workflow=vibe-screenshots.yml --limit 1 --json databaseId --jq '.[0].databaseId')+gh run watch $RUN_ID```-internal/vibe-tests/results/<iteration>/runs.jsonl++Download artifacts:++```bash+gh run download $RUN_ID -n vibe-test-screenshots -D internal/vibe-tests/results/+gh run download $RUN_ID -n preview-html -D internal/vibe-tests/results/```-Each line should be a JSON object with:--```json-{- "id": "<iteration>-<promptId>",- "timestamp": "<ISO timestamp>",- "systemVersion": "xds-llms-v1",- "model": "claude-code-interactive",- "persona": "naive",- "promptCategory": "<category>",- "trajectoryDepth": 0,- "prompt": "<the prompt>",- "response": "<generated code>",- "evaluation": {- "success": true/false,- "componentsUsed": ["XDSCard", ...],- "componentsExpected": ["XDSCard", ...],- "escapeHatches": [{"type": "...", "severity": "acceptable/critical", "detail": "...", "codeSnippet": "..."}],- "failureMode": null or "description",- "confusionSignals": []- },- "fullConversation": [],- "contextWindowUsage": 0,- "durationMs": <milliseconds taken>,- "inputTokens": <input tokens used>,- "outputTokens": <output tokens generated>-}+If GHA fails: `gh run view $RUN_ID --log-failed`++### Step 6: Aggregate and report++```bash+cd internal/vibe-tests+yarn aggregate --iteration <xds-id>+yarn report:deploy --iteration <xds-id> --baseline <baseline-id> --html <html-id>```-### Step 5: Aggregate results+Report URL: `https://facebookexperimental.github.io/xds/reports/<xds-id>/`++### Step 7: Post results-Run:+Create a GitHub issue:```bash-yarn workspace @xds/vibe-tests aggregate --iteration <id>+gh issue create --repo facebookexperimental/xds \+ --title "Vibe Test Results — $(date +%Y-%m-%d)" \+ --label "vibe-test" \+ --body "<results summary>"```-This will print a summary showing:+Include: overall scores, dimension winners, report link, gaps, and screenshots.-- Overall success rate-- Performance stats (total time, avg time, tokens used)-- Success by category-- Critical issues (if any)-- Acceptable escape hatches+## Fairness Rules-It also generates an HTML report at `results/<iteration>/report.html` that you can open in a browser to inspect individual test results.+These rules ensure the comparison is meaningful:-### Evaluation Criteria+1. **Same prompts** — all targets run the exact same prompt IDs (use `--prompts`)+2. **Same persona** — always `naive` (simulates a developer with no prior knowledge)+3. **Fair docs** — each target gets its own equivalent agent doc, not the other target's+4. **No cross-contamination** — XDS agents don't see baseline docs, and vice versa+5. **10 prompts minimum** — stratified sample covers all categories-**Quality Tiers:**+## Evaluation Criteria-- 🥇 **Gold** - Pure XDS, no escape hatches needed-- 🟢 **Green** - Correct components, only acceptable escape hatches-- 🟡 **Yellow** - Anti-patterns that break theming (hardcoded values)-- 🔴 **Red** - Critical failures+### Quality Tiers-**Critical escape hatches (Red tier):**+- 🥇 **Gold** — Pure target usage, no escape hatches needed+- 🟢 **Green** — Correct components, only acceptable escape hatches+- 🟡 **Yellow** — Anti-patterns that break theming (hardcoded values)+- 🔴 **Red** — Critical failures (hallucinations, wrong components)-- `hallucination` - Invented props/components that don't exist-- `wrong_component` - Using a component incorrectly-- `redundant_css` - CSS that duplicates a component prop+### Escape Hatch Classification-**Anti-pattern escape hatches (Yellow tier - break theming):**+**Critical (Red tier):**+- `hallucination` — invented props/components that don't exist+- `wrong_component` — using a component incorrectly+- `redundant_css` — CSS that duplicates a component prop-- `hardcoded_color` - Direct color values instead of CSS variables-- `hardcoded_spacing` - Direct pixel values for spacing instead of CSS variables-- `hardcoded_size` - Direct pixel values for sizes instead of CSS variables-- `inline_style` - Inline styles instead of StyleX+**Anti-pattern (Yellow tier):**+- `hardcoded_color` — direct color values instead of CSS variables+- `hardcoded_spacing` — direct pixel values instead of CSS variables+- `inline_style` — inline styles instead of StyleX-**Acceptable escape hatches (Green tier):**+**Acceptable (Green tier):**+- `supplemental_css` — StyleX for things the system doesn't cover+- `wrapper_div` — structural HTML for composition+- `custom_animation` — animation not covered by the system-- `supplemental_css` - StyleX for things XDS doesn't cover-- `wrapper_div` - Structural HTML for composition-- `custom_animation` - Animation not covered by XDS-- `layout_workaround` - Layout patterns XDS doesn't support+## Evaluation Dimensions-When evaluating, also include a `gap` field on escape hatches describing what capability is missing, to help generate improvement suggestions.+| Dimension | What It Measures |+|-----------|-----------------|+| Correctness | Valid component usage, no hallucinations |+| Accessibility | Labels, semantics, keyboard support |+| Code Quality | Complexity, patterns, TypeScript usage |+| Efficiency | Decisions/element, DRYness, conciseness |+| Maintainability | Semantic tokens vs magic values, dark mode, locality |diff --git a/.github/workflows/vibe-screenshots.yml b/.github/workflows/vibe-screenshots.ymlnew file mode 100644index 0000000000..59e03665c9--- /dev/null+++ b/.github/workflows/vibe-screenshots.yml@@ -0,0 +1,100 @@+name: Vibe Test Screenshots++on:+ workflow_dispatch:+ inputs:+ iterations:+ description: 'Comma-separated iteration IDs to screenshot'+ required: true+ prompts:+ description: 'Comma-separated prompt IDs (optional, defaults to all)'+ required: false+ default: ''++concurrency:+ group: vibe-screenshots-${{ github.run_id }}+ cancel-in-progress: false++jobs:+ build-previews:+ name: Build Previews+ runs-on: ubuntu-latest+ steps:+ - name: Checkout+ uses: actions/checkout@v4++ - name: Setup Node.js+ uses: actions/setup-node@v4+ with:+ node-version: '22'+ cache: 'yarn'++ - name: Install dependencies+ run: yarn install --frozen-lockfile++ - name: Build core packages+ run: yarn build++ - name: Build preview HTML files+ run: |+ cd internal/vibe-tests+ PROMPT_ARG=""+ if [ -n "${{ github.event.inputs.prompts }}" ]; then+ PROMPT_ARG="--prompts ${{ github.event.inputs.prompts }}"+ fi+ npx tsx src/build-previews.ts \+ --iterations "${{ github.event.inputs.iterations }}" \+ $PROMPT_ARG++ - name: Upload preview HTML+ uses: actions/upload-artifact@v4+ with:+ name: preview-html+ path: internal/vibe-tests/results/*/previews/+ retention-days: 90++ screenshot:+ name: Capture Screenshots+ needs: build-previews+ runs-on: ubuntu-latest+ steps:+ - name: Checkout+ uses: actions/checkout@v4++ - name: Setup Node.js+ uses: actions/setup-node@v4+ with:+ node-version: '22'+ cache: 'yarn'++ - name: Install dependencies+ run: yarn install --frozen-lockfile++ - name: Install Playwright+ run: npx playwright install chromium --with-deps++ - name: Download preview HTML+ uses: actions/download-artifact@v4+ with:+ name: preview-html+ path: internal/vibe-tests/results/++ - name: Screenshot all previews+ run: |+ cd internal/vibe-tests+ npx tsx src/screenshot-previews.ts \+ --iterations "${{ github.event.inputs.iterations }}"++ - name: Upload screenshots+ uses: actions/upload-artifact@v4+ with:+ name: vibe-test-screenshots+ path: internal/vibe-tests/results/*/screenshots/+ retention-days: 90++ - name: Upload screenshot manifest+ uses: actions/upload-artifact@v4+ with:+ name: screenshot-manifest+ path: internal/vibe-tests/results/*/screenshots/manifest.json+ retention-days: 90diff --git a/.github/workflows/vibe-tests.yml.disabled b/.github/workflows/vibe-tests.yml.disableddeleted file mode 100644index f98ccc0943..0000000000--- a/.github/workflows/vibe-tests.yml.disabled+++ /dev/null@@ -1,224 +0,0 @@-# =============================================================================-# DISABLED: This workflow is temporarily disabled.-#-# To re-enable:-# 1. Add ANTHROPIC_API_KEY secret to the repository-# 2. Rename this file back to vibe-tests.yml-# =============================================================================--name: Vibe Tests--on:- pull_request:- branches: [main, master]- push:- branches: [main, master]- workflow_dispatch:- inputs:- sample_size:- description: 'Number of test prompts to run'- required: false- default: '10'--concurrency:- group: vibe-tests-${{ github.head_ref || github.ref }}- cancel-in-progress: true--jobs:- vibe-test:- name: Run Vibe Tests- runs-on: ubuntu-latest- permissions:- contents: read- pull-requests: write- actions: read-- steps:- - name: Checkout- uses: actions/checkout@v4-- - name: Setup Node.js- uses: actions/setup-node@v4- with:- node-version: '22'- cache: 'yarn'-- - name: Install dependencies- run: yarn install --frozen-lockfile-- - name: Run vibe tests- id: vibe-test- env:- ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}- SKILL_DOC: packages/core/xds.md- run: |- # Determine sample size- SAMPLE_SIZE="${{ github.event.inputs.sample_size || '10' }}"-- # Run the tests- echo "Running vibe tests with sample size: $SAMPLE_SIZE"- yarn workspace @xds/vibe-tests test:xds:sample 2>&1 | tee vibe-test-output.txt-- # Extract iteration ID from output- ITERATION_ID=$(grep -oP 'Iteration: \K[a-f0-9]+' vibe-test-output.txt | head -1)- echo "iteration_id=$ITERATION_ID" >> $GITHUB_OUTPUT- echo "Iteration ID: $ITERATION_ID"-- - name: Aggregate results- id: aggregate- if: steps.vibe-test.outputs.iteration_id != ''- run: |- ITERATION_ID="${{ steps.vibe-test.outputs.iteration_id }}"- yarn workspace @xds/vibe-tests aggregate --iteration "$ITERATION_ID" --ci-- - name: Upload results artifact- if: steps.vibe-test.outputs.iteration_id != ''- uses: actions/upload-artifact@v4- with:- name: vibe-test-results-${{ steps.vibe-test.outputs.iteration_id }}- path: |- internal/vibe-tests/results/${{ steps.vibe-test.outputs.iteration_id }}/- retention-days: 90-- - name: Upload HTML report- if: steps.vibe-test.outputs.iteration_id != ''- uses: actions/upload-artifact@v4- with:- name: vibe-test-report-${{ steps.vibe-test.outputs.iteration_id }}- path: internal/vibe-tests/results/${{ steps.vibe-test.outputs.iteration_id }}/report.html- retention-days: 90-- - name: Comment on PR- if: github.event_name == 'pull_request' && steps.vibe-test.outputs.iteration_id != ''- uses: actions/github-script@v7- with:- script: |- const successRate = '${{ steps.aggregate.outputs.success_rate }}';- const gold = '${{ steps.aggregate.outputs.gold }}' || '0';- const green = '${{ steps.aggregate.outputs.green }}' || '0';- const yellow = '${{ steps.aggregate.outputs.yellow }}' || '0';- const red = '${{ steps.aggregate.outputs.red }}' || '0';- const iterationId = '${{ steps.vibe-test.outputs.iteration_id }}';- const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;-- // Determine status emoji- let statusEmoji = '🔴';- if (parseInt(successRate) >= 90) statusEmoji = '🟢';- else if (parseInt(successRate) >= 70) statusEmoji = '🟡';-- const body = `## ${statusEmoji} Vibe Test Results-- **Iteration:** \`${iterationId}\`- **Success Rate:** ${successRate}%-- ### Quality Tiers- | Tier | Count |- |------|-------|- | 🥇 Gold (pure XDS) | ${gold} |- | 🟢 Green (acceptable) | ${green} |- | 🟡 Yellow (anti-pattern) | ${yellow} |- | 🔴 Red (critical) | ${red} |-- <details>- <summary>Tier Definitions</summary>-- - **Gold**: Pure XDS, no escape hatches needed- - **Green**: Correct components with acceptable escape hatches- - **Yellow**: Anti-patterns that break theming (hardcoded values, inline styles)- - **Red**: Critical failures (hallucinations, wrong components)- </details>-- 📊 [View full report](${runUrl}) (download artifacts)- `;-- // Find existing comment- const { data: comments } = await github.rest.issues.listComments({- owner: context.repo.owner,- repo: context.repo.repo,- issue_number: context.issue.number,- });-- const botComment = comments.find(comment =>- comment.user.type === 'Bot' &&- comment.body.includes('Vibe Test Results')- );-- if (botComment) {- await github.rest.issues.updateComment({- owner: context.repo.owner,- repo: context.repo.repo,- comment_id: botComment.id,- body: body- });- } else {- await github.rest.issues.createComment({- owner: context.repo.owner,- repo: context.repo.repo,- issue_number: context.issue.number,- body: body- });- }-- - name: Check threshold- if: steps.aggregate.outputs.success_rate != ''- run: |- SUCCESS_RATE="${{ steps.aggregate.outputs.success_rate }}"- RED="${{ steps.aggregate.outputs.red }}"-- echo "✅ Success rate: $SUCCESS_RATE%"- echo "✅ Critical failures: $RED"-- # Fail if success rate is below 70%- if [ "$SUCCESS_RATE" -lt 70 ]; then- echo "::error::Vibe test success rate ($SUCCESS_RATE%) is below threshold (70%)"- exit 1- fi-- echo "✅ Vibe tests passed with $SUCCESS_RATE% success rate"-- store-results:- name: Store Historical Results- needs: vibe-test- if: github.event_name == 'push' && github.ref == 'refs/heads/main'- runs-on: ubuntu-latest- permissions:- contents: write-- steps:- - name: Checkout- uses: actions/checkout@v4- with:- ref: vibe-test-history- fetch-depth: 0-- - name: Download artifacts- uses: actions/download-artifact@v4- with:- pattern: vibe-test-results-*- path: results/- merge-multiple: true-- - name: Commit results- run: |- git config user.name "github-actions[bot]"- git config user.email "github-actions[bot]@users.noreply.github.com"-- # Create branch if it doesn't exist- if ! git rev-parse --verify vibe-test-history >/dev/null 2>&1; then- git checkout --orphan vibe-test-history- git rm -rf . || true- echo "# Vibe Test Results History" > README.md- echo "" >> README.md- echo "This branch stores historical vibe test results from CI runs." >> README.md- fi-- # Add new results- TIMESTAMP=$(date +%Y-%m-%d-%H%M%S)- COMMIT_SHA="${{ github.sha }}"- mkdir -p "history/${TIMESTAMP}-${COMMIT_SHA:0:7}"- cp -r results/* "history/${TIMESTAMP}-${COMMIT_SHA:0:7}/" || true-- # Commit and push- git add .- git commit -m "Add vibe test results from ${COMMIT_SHA:0:7}" || echo "No changes to commit"- git push origin vibe-test-history || echo "Failed to push, branch may not exist"diff --git a/CLAUDE.md b/CLAUDE.mdindex b181ba85b3..ca5d794d61 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -18,13 +18,15 @@ Tests how well AGENTS.md helps LLMs generate correct XDS component code.**How to execute:**-1. Run `yarn workspace @xds/vibe-tests interactive --sample <count> [--degradation]` to set up iteration-2. Spawn parallel subagents (one per test prompt) with `mode: "bypassPermissions"` to:+1. Run `yarn workspace @xds/vibe-tests interactive --sample <count>` to set up iteration+2. Spawn parallel subagents (one per test prompt) to:- Read the task file from `results/<iteration>/tasks/{promptId}.json`- Generate code for the prompt using XDS components (AGENTS.md auto-injected)- Self-evaluate for success/escape hatches- - Write result to individual file: `results/<iteration>/results/{promptId}.json`-3. Run `yarn workspace @xds/vibe-tests aggregate --iteration <id>` to see results+ - Write `.tsx` result to `results/<iteration>/results/{promptId}.tsx`+ - Write `.json` metadata to `results/<iteration>/results/{promptId}.json`+3. Trigger `gh workflow run vibe-screenshots.yml` to build previews and capture screenshots+4. Run `yarn workspace @xds/vibe-tests aggregate --iteration <id>` to see results**Degradation mode (--degradation):**Tests context retention across 10-turn conversations with filler, distractor, and recovery turns.diff --git a/internal/vibe-tests/README.md b/internal/vibe-tests/README.mdindex 49baa3bbce..11664a680a 100644--- a/internal/vibe-tests/README.md+++ b/internal/vibe-tests/README.md@@ -41,8 +41,13 @@ The skill doc is generated fresh each time, so it always reflects the current br## Quick Start - API Mode```bash-# Requires ANTHROPIC_API_KEY+# Requires ANTHROPIC_API_KEY (direct Anthropic mode)yarn workspace @xds/vibe-tests test:xds:sample++# Or use the Navi-orchestrated pipeline (no API key needed):+# 1. Navi sub-agents generate TSX results+# 2. GHA builds previews and captures screenshots+# See: .github/workflows/vibe-screenshots.yml```## What It Does@@ -234,29 +239,25 @@ Suggested refinements:## CI Integration-Vibe tests run automatically on every PR via GitHub Actions.+### Screenshot Service (`.github/workflows/vibe-screenshots.yml`)-### GitHub Action+The primary workflow for nightly vibe tests. Triggered by Navi via `workflow_dispatch`:-The workflow (`.github/workflows/vibe-tests.yml`) runs on:+```bash+gh workflow run vibe-screenshots.yml -f iterations="<id1>,<id2>,<id3>"+```-- Every pull request to main/master-- Every push to main/master-- Manual trigger with configurable sample size+**What it does:**-### What it does:+1. Builds self-contained HTML previews from TSX results (inlines all CSS via vite-plugin-singlefile)+2. Captures Playwright screenshots at 4 combos: desktop/mobile × light/dark+3. Uploads preview HTML and screenshots as artifacts (90-day retention)-1. Runs 10 stratified sample tests-2. Aggregates results and generates HTML report-3. Posts a summary comment on the PR-4. Uploads results as artifacts (90-day retention)-5. Fails the check if:- - Success rate is below 70%- - Any critical (Red tier) failures+**No API keys required.** Code generation is handled by Navi sub-agents before this workflow runs.-### Required Secrets:+### Direct API Mode (legacy)-- `ANTHROPIC_API_KEY`: API key for running tests+The `test:xds:sample` script calls Anthropic directly and requires `ANTHROPIC_API_KEY`. This is the old all-in-one flow — prefer the Navi-orchestrated pipeline for nightly runs.### Aggregate CLI Flags:diff --git a/internal/vibe-tests/package.json b/internal/vibe-tests/package.jsonindex 6a49b47ee7..a2a8951a78 100644--- a/internal/vibe-tests/package.json+++ b/internal/vibe-tests/package.json@@ -10,10 +10,6 @@"scripts": {"harness": "tsx src/cli.ts","generate:skill-doc": "bash scripts/generate-skill-doc.sh",- "test:xds": "yarn generate:skill-doc && SKILL_DOC=.generated/xds-skill.md tsx src/cli.ts run",- "test:xds:sample": "yarn generate:skill-doc && SKILL_DOC=.generated/xds-skill.md tsx src/cli.ts run --sample 10",- "test:xds:holdout": "yarn generate:skill-doc && SKILL_DOC=.generated/xds-skill.md tsx src/cli.ts run --holdout",- "test:ci": "yarn generate:skill-doc && SKILL_DOC=.generated/xds-skill.md tsx src/cli.ts run --sample 10","interactive": "tsx src/interactive.ts","interactive:sample": "tsx src/interactive.ts --sample 5","run-interactive": "tsx src/run-interactive.ts",@@ -21,7 +17,6 @@"aggregate": "tsx src/aggregate.ts","aggregate:ci": "tsx src/aggregate.ts --ci","aggregate:json": "tsx src/aggregate.ts --json",- "analyze": "tsx src/cli.ts analyze","quality": "tsx src/quality.ts","compare": "tsx src/compare.ts","generate-a11y-manifest": "tsx src/generate-a11y-manifest.ts",@@ -31,10 +26,10 @@"report:dev": "tsx src/build-report.ts --dev","report:deploy": "tsx src/deploy-report.ts","preview:dev": "vite --config app/vite.config.preview.ts",- "preview:build": "vite build --config app/vite.config.preview.ts"+ "preview:build": "vite build --config app/vite.config.preview.ts",+ "screenshot:previews": "tsx src/screenshot-previews.ts"},"dependencies": {- "@anthropic-ai/sdk": "^0.52.0","@xds/cli": "*","commander": "^12.1.0"},diff --git a/internal/vibe-tests/src/analyst.ts b/internal/vibe-tests/src/analyst.tsdeleted file mode 100644index 4a0c30e7f5..0000000000--- a/internal/vibe-tests/src/analyst.ts+++ /dev/null@@ -1,101 +0,0 @@-/**- * @file Analyst module - analyzes test results and suggests refinements- * @position internal/vibe-tests/src/analyst.ts- */--import Anthropic from '@anthropic-ai/sdk';-import * as path from 'node:path';-import type {TestResult, AnalysisResult} from './types.js';-import {readPromptFile, getPromptsDir} from './utils.js';--const ANALYST_PROMPT_PATH = path.join(getPromptsDir(), 'analyst.md');--export interface AnalyzeInput {- results: TestResult[];- skillDoc: string;- promptsRun: number;- promptsTotal: number;-}--/**- * Analyze a batch of test results and suggest refinements- */-export async function analyze(- client: Anthropic,- input: AnalyzeInput,- model: string,-): Promise<AnalysisResult> {- const analystPrompt = readPromptFile(ANALYST_PROMPT_PATH);-- const systemPrompt = `${analystPrompt}--## Skill Doc (Component System Documentation)--${input.skillDoc}`;-- // Prepare results summary (full results could be too large)- const resultsSummary = input.results.map(r => ({- id: r.id,- category: r.promptCategory,- persona: r.persona,- prompt: r.prompt,- success: r.evaluation.success,- componentsUsed: r.evaluation.componentsUsed,- componentsExpected: r.evaluation.componentsExpected,- escapeHatches: r.evaluation.escapeHatches,- failureMode: r.evaluation.failureMode,- confusionSignals: r.evaluation.confusionSignals,- trajectoryDepth: r.trajectoryDepth,- }));-- const userMessage = `## Sampling Metadata--- Prompts run: ${input.promptsRun}-- Total prompts in pool: ${input.promptsTotal}-- Sampling: ${input.promptsRun < input.promptsTotal ? 'Yes (stratified)' : 'No (full run)'}--## Test Results--\`\`\`json-${JSON.stringify(resultsSummary, null, 2)}-\`\`\`-------Analyze these results and return JSON matching the AnalysisResult schema. Only output valid JSON, no other text.`;-- const response = await client.messages.create({- model,- max_tokens: 4096,- system: systemPrompt,- messages: [{role: 'user', content: userMessage}],- });-- const content = response.content[0];- if (content.type !== 'text') {- throw new Error('Expected text response from analyst');- }-- // Parse JSON from response- let jsonText = content.text.trim();- if (jsonText.startsWith('```json')) {- jsonText = jsonText.slice(7);- } else if (jsonText.startsWith('```')) {- jsonText = jsonText.slice(3);- }- if (jsonText.endsWith('```')) {- jsonText = jsonText.slice(0, -3);- }- jsonText = jsonText.trim();-- try {- return JSON.parse(jsonText) as AnalysisResult;- } catch {- console.error('Failed to parse analyst response:', content.text);- return {- patterns: [],- refinements: [],- summary: 'Failed to parse analyst response',- };- }-}diff --git a/internal/vibe-tests/src/cli.ts b/internal/vibe-tests/src/cli.tsindex d45d1c8adb..9f94e7a854 100755--- a/internal/vibe-tests/src/cli.ts+++ b/internal/vibe-tests/src/cli.ts@@ -2,19 +2,18 @@/*** @file CLI entry point for vibe-tests harness* @position internal/vibe-tests/src/cli.ts+ *+ * Code generation is handled by Navi sub-agents (no API key needed).+ * This CLI provides aggregation, reporting, and utility commands.*/import {Command} from 'commander';import * as path from 'node:path';import * as fs from 'node:fs';-import Anthropic from '@anthropic-ai/sdk';-import type {RunConfig, PersonaType, Protocol, Iteration} from './types.js';-import {runTests} from './runner.js';-import {analyze} from './analyst.js';+import type {Iteration} from './types.js';import {loadTestSet,readJsonl,- writeJson,readJson,getResultsDir,} from './utils.js';@@ -27,162 +26,6 @@ program.description('Vibeability test harness for LLM component system usage').version('0.0.1');-/**- * Run command- */-program- .command('run')- .description('Run vibeability tests')- .option('--protocol <type>', 'Test protocol', 'one-shot')- .option('--persona <type>', 'User persona', 'naive')- .option('--model <model>', 'LLM model', 'claude-sonnet-4-20250514')- .option('--skill-doc <path>', 'Path to skill doc')- .option('--sample <n>', 'Sample N prompts (stratified)', parseInt)- .option('--holdout', 'Run holdout set instead of main prompts')- .action(async options => {- const skillDocPath =- options.skillDoc ?? process.env.SKILL_DOC ?? './skills/brownie.md';-- if (!fs.existsSync(skillDocPath)) {- console.error(`Error: Skill doc not found at ${skillDocPath}`);- console.error(- 'Set SKILL_DOC env var or use --skill-doc flag to specify path',- );- process.exit(1);- }-- const config: RunConfig = {- protocol: options.protocol as Protocol,- persona: options.persona as PersonaType,- model: options.model,- skillDocPath,- sample: options.sample,- holdout: options.holdout,- };-- const testSet = loadTestSet('default');- const result = await runTests(testSet, config);-- // Print results summary- console.log('Results:');- console.log(- ` Overall: ${calculateSuccessRate(result.results)}% success (${result.results.filter(r => r.evaluation.success).length}/${result.results.length})\n`,- );-- // By category- console.log(' By category:');- const byCategory = groupBy(result.results, r => r.promptCategory);- for (const [category, results] of Object.entries(byCategory)) {- console.log(` ${category}: ${calculateSuccessRate(results)}%`);- }-- // Escape hatches- console.log('\n Critical issues:');- const criticalHatches = result.results.flatMap(r =>- r.evaluation.escapeHatches.filter(- (h): h is import('./types.js').EscapeHatch =>- typeof h !== 'string' && h.severity === 'critical',- ),- );- const criticalCounts = countBy(criticalHatches, h => h.type);- for (const [type, count] of Object.entries(criticalCounts)) {- console.log(` - ${type} (${count})`);- }-- console.log('\n Acceptable escape hatches:');- const acceptableHatches = result.results.flatMap(r =>- r.evaluation.escapeHatches.filter(- (h): h is import('./types.js').EscapeHatch =>- typeof h !== 'string' && h.severity === 'acceptable',- ),- );- const acceptableCounts = countBy(acceptableHatches, h => h.type);- for (const [type, count] of Object.entries(acceptableCounts)) {- console.log(` - ${type} (${count})`);- }-- console.log(- `\n Results saved: ${path.join(getResultsDir(), result.iterationId, 'runs.jsonl')}`,- );- console.log(- `\nRun \`vibe-tests analyze --iteration ${result.iterationId}\` for refinement suggestions.`,- );- });--/**- * Analyze command- */-program- .command('analyze')- .description('Analyze test results from an iteration')- .requiredOption('--iteration <id>', 'Iteration ID to analyze')- .option(- '--model <model>',- 'LLM model for analysis',- 'claude-sonnet-4-20250514',- )- .option('--skill-doc <path>', 'Path to skill doc')- .action(async options => {- const iterationId = options.iteration;- const resultsDir = path.join(getResultsDir(), iterationId);-- if (!fs.existsSync(resultsDir)) {- console.error(`Error: Iteration ${iterationId} not found`);- process.exit(1);- }-- const runsPath = path.join(resultsDir, 'runs.jsonl');- const results = readJsonl<TestResult>(runsPath);-- if (results.length === 0) {- console.error(`Error: No results found for iteration ${iterationId}`);- process.exit(1);- }-- const skillDocPath =- options.skillDoc ??- process.env.SKILL_DOC ??- `./skills/${results[0].systemVersion}.md`;-- if (!fs.existsSync(skillDocPath)) {- console.error(`Error: Skill doc not found at ${skillDocPath}`);- process.exit(1);- }-- const skillDoc = fs.readFileSync(skillDocPath, 'utf-8');- const promptsRun = new Set(results.map(r => r.id.split('-')[1])).size;- const promptsTotal = loadTestSet('default').prompts.length;-- console.log(`\nAnalyzing iteration ${iterationId}...`);-- const client = new Anthropic();- const analysis = await analyze(- client,- {results, skillDoc, promptsRun, promptsTotal},- options.model,- );-- // Save analysis- const analysisPath = path.join(resultsDir, 'analysis.json');- writeJson(analysisPath, analysis);-- // Print results- console.log('\nPatterns detected:');- for (const pattern of analysis.patterns) {- console.log(` - ${pattern}`);- }-- console.log('\nSuggested refinements:');- for (let i = 0; i < analysis.refinements.length; i++) {- const ref = analysis.refinements[i];- console.log(- ` ${i + 1}. [${ref.target}] ${ref.suggestion} (confidence: ${ref.confidence.toFixed(1)}, effort: ${ref.effortEstimate})`,- );- }-- console.log(`\nSaved: ${analysisPath}`);- });-/*** Report command*/@@ -227,35 +70,4 @@ program}});-// Helper functions-function calculateSuccessRate(results: TestResult[]): string {- const successes = results.filter(r => r.evaluation.success).length;- return ((successes / results.length) * 100).toFixed(0);-}--function groupBy<T>(- items: T[],- keyFn: (item: T) => string,-): Record<string, T[]> {- const groups: Record<string, T[]> = {};- for (const item of items) {- const key = keyFn(item);- if (!groups[key]) groups[key] = [];- groups[key].push(item);- }- return groups;-}--function countBy<T>(- items: T[],- keyFn: (item: T) => string,-): Record<string, number> {- const counts: Record<string, number> = {};- for (const item of items) {- const key = keyFn(item);- counts[key] = (counts[key] ?? 0) + 1;- }- return counts;-}-program.parse();diff --git a/internal/vibe-tests/src/evaluator.ts b/internal/vibe-tests/src/evaluator.tsdeleted file mode 100644index 203f942bb8..0000000000--- a/internal/vibe-tests/src/evaluator.ts+++ /dev/null@@ -1,90 +0,0 @@-/**- * @file Evaluator module - judges LLM responses against component system- * @position internal/vibe-tests/src/evaluator.ts- */--import Anthropic from '@anthropic-ai/sdk';-import * as path from 'node:path';-import type {Evaluation} from './types.js';-import {readPromptFile, getPromptsDir} from './utils.js';--const EVALUATOR_PROMPT_PATH = path.join(getPromptsDir(), 'evaluator.md');--export interface EvaluateInput {- userPrompt: string;- llmResponse: string;- expectedComponents: string[];- skillDoc: string;-}--/**- * Evaluate an LLM response against the component system- */-export async function evaluate(- client: Anthropic,- input: EvaluateInput,- model: string,-): Promise<Evaluation> {- const evaluatorPrompt = readPromptFile(EVALUATOR_PROMPT_PATH);-- const systemPrompt = `${evaluatorPrompt}--## Skill Doc (Component System Documentation)--${input.skillDoc}`;-- const userMessage = `## User's Original Prompt--${input.userPrompt}--## Expected Components--${input.expectedComponents.join(', ')}--## LLM Response to Evaluate--${input.llmResponse}-------Evaluate this response and return JSON matching the Evaluation schema. Only output valid JSON, no other text.`;-- const response = await client.messages.create({- model,- max_tokens: 2048,- system: systemPrompt,- messages: [{role: 'user', content: userMessage}],- });-- const content = response.content[0];- if (content.type !== 'text') {- throw new Error('Expected text response from evaluator');- }-- // Parse JSON from response, handling potential markdown code blocks- let jsonText = content.text.trim();- if (jsonText.startsWith('```json')) {- jsonText = jsonText.slice(7);- } else if (jsonText.startsWith('```')) {- jsonText = jsonText.slice(3);- }- if (jsonText.endsWith('```')) {- jsonText = jsonText.slice(0, -3);- }- jsonText = jsonText.trim();-- try {- return JSON.parse(jsonText) as Evaluation;- } catch {- // If parsing fails, return a failure evaluation- console.error('Failed to parse evaluator response:', content.text);- return {- success: false,- componentsUsed: [],- componentsExpected: input.expectedComponents,- escapeHatches: [],- failureMode: 'evaluator_parse_error',- confusionSignals: [],- };- }-}diff --git a/internal/vibe-tests/src/index.ts b/internal/vibe-tests/src/index.tsindex b956cf2388..0fca89e461 100644--- a/internal/vibe-tests/src/index.ts+++ b/internal/vibe-tests/src/index.ts@@ -4,7 +4,4 @@*/export * from './types.js';-export {runTests} from './runner.js';-export {evaluate} from './evaluator.js';-export {analyze} from './analyst.js';export {stratifiedSample} from './sampling.js';diff --git a/internal/vibe-tests/src/runner.ts b/internal/vibe-tests/src/runner.tsdeleted file mode 100644index 71233b025c..0000000000--- a/internal/vibe-tests/src/runner.ts+++ /dev/null@@ -1,374 +0,0 @@-/**- * @file Test runner - orchestrates test execution- * @position internal/vibe-tests/src/runner.ts- */--import Anthropic from '@anthropic-ai/sdk';-import * as fs from 'node:fs';-import * as path from 'node:path';-import type {- TestPrompt,- TestResult,- TestSet,- Turn,- RunConfig,- PersonaType,- Evaluation,-} from './types.js';-import {evaluate} from './evaluator.js';-import {stratifiedSample} from './sampling.js';-import {- generateIterationId,- hashContent,- estimateTokens,- sleep,- ensureDir,- appendJsonl,- readPromptFile,- getPromptsDir,- getResultsDir,- progressBar,- timestamp,-} from './utils.js';--const RATE_LIMIT_DELAY = 500; // ms between requests--export interface RunResult {- iterationId: string;- results: TestResult[];- promptsRun: string[];- promptsTotal: number;-}--/**- * Load persona prompt- */-function loadPersona(persona: PersonaType): string {- const personaPath = path.join(getPromptsDir(), 'personas', `${persona}.md`);- return readPromptFile(personaPath);-}--/**- * Wrap user prompt with persona framing- */-function applyPersona(prompt: string, personaPrompt: string): string {- return `${personaPrompt}-------Now, as this persona, make the following request:--${prompt}`;-}--/**- * Run a single test prompt (one-shot)- */-async function runOneShot(- client: Anthropic,- testPrompt: TestPrompt,- skillDoc: string,- personaPrompt: string,- config: RunConfig,-): Promise<{response: string; conversation: Turn[]; tokenUsage: number}> {- const framedPrompt = applyPersona(testPrompt.prompt, personaPrompt);-- const response = await client.messages.create({- model: config.model,- max_tokens: 4096,- system: skillDoc,- messages: [{role: 'user', content: framedPrompt}],- });-- const content = response.content[0];- if (content.type !== 'text') {- throw new Error('Expected text response');- }-- const turn: Turn = {- type: 'probe',- prompt: testPrompt.prompt,- response: content.text,- };-- const tokenUsage =- estimateTokens(skillDoc) +- estimateTokens(framedPrompt) +- estimateTokens(content.text);-- return {- response: content.text,- conversation: [turn],- tokenUsage,- };-}--/**- * Run degradation curve test- */-async function runDegradation(- client: Anthropic,- testPrompt: TestPrompt,- skillDoc: string,- personaPrompt: string,- fillerPrompts: string[],- distractorPrompts: string[],- recoveryContext: string,- config: RunConfig,-): Promise<{- turns: Turn[];- evaluations: Map<number, Evaluation>;- tokenUsage: number;-}> {- const messages: Anthropic.MessageParam[] = [];- const turns: Turn[] = [];- const evaluations = new Map<number, Evaluation>();- let tokenUsage = estimateTokens(skillDoc);-- // Helper to add a turn- const addTurn = async (- type: Turn['type'],- prompt: string,- ): Promise<string> => {- const framedPrompt =- type === 'probe' ? applyPersona(prompt, personaPrompt) : prompt;-- messages.push({role: 'user', content: framedPrompt});- tokenUsage += estimateTokens(framedPrompt);-- await sleep(RATE_LIMIT_DELAY);- const response = await client.messages.create({- model: config.model,- max_tokens: 4096,- system: skillDoc,- messages,- });-- const content = response.content[0];- if (content.type !== 'text') {- throw new Error('Expected text response');- }-- messages.push({role: 'assistant', content: content.text});- tokenUsage += estimateTokens(content.text);-- turns.push({type, prompt, response: content.text});- return content.text;- };-- // Turn 0: Initial probe- const initialResponse = await addTurn('probe', testPrompt.prompt);- const eval0 = await evaluate(- client,- {- userPrompt: testPrompt.prompt,- llmResponse: initialResponse,- expectedComponents: testPrompt.expectedComponents,- skillDoc,- },- config.model,- );- turns[0].evaluation = eval0;- evaluations.set(0, eval0);-- // Turns 1-5: Filler- for (let i = 0; i < 5; i++) {- const fillerPrompt =- fillerPrompts[Math.floor(Math.random() * fillerPrompts.length)];- await addTurn('filler', fillerPrompt);- }-- // Turn 6: Re-probe with natural framing- const reprobe1 = `Actually, let me revisit this. ${testPrompt.prompt}`;- const response6 = await addTurn('probe', reprobe1);- const eval6 = await evaluate(- client,- {- userPrompt: testPrompt.prompt,- llmResponse: response6,- expectedComponents: testPrompt.expectedComponents,- skillDoc,- },- config.model,- );- turns[turns.length - 1].evaluation = eval6;- evaluations.set(6, eval6);-- // Turn 7: Distractor- const distractor =- distractorPrompts[Math.floor(Math.random() * distractorPrompts.length)];- await addTurn('distractor', distractor);-- // Turn 8: Re-probe after distractor- const reprobe2 = `I want to redo that earlier request. ${testPrompt.prompt}`;- const response8 = await addTurn('probe', reprobe2);- const eval8 = await evaluate(- client,- {- userPrompt: testPrompt.prompt,- llmResponse: response8,- expectedComponents: testPrompt.expectedComponents,- skillDoc,- },- config.model,- );- turns[turns.length - 1].evaluation = eval8;- evaluations.set(8, eval8);-- // Turn 9: Recovery (re-inject partial skill doc)- const recovery = `${recoveryContext}\n\n${skillDoc.slice(0, 2000)}...`;- await addTurn('recovery', recovery);-- // Turn 10: Final re-probe after recovery- const reprobe3 = `Let me try that again. ${testPrompt.prompt}`;- const response10 = await addTurn('probe', reprobe3);- const eval10 = await evaluate(- client,- {- userPrompt: testPrompt.prompt,- llmResponse: response10,- expectedComponents: testPrompt.expectedComponents,- skillDoc,- },- config.model,- );- turns[turns.length - 1].evaluation = eval10;- evaluations.set(10, eval10);-- return {turns, evaluations, tokenUsage};-}--/**- * Run tests with the given configuration- */-export async function runTests(- testSet: TestSet,- config: RunConfig,-): Promise<RunResult> {- const client = new Anthropic();- const iterationId = generateIterationId();- const skillDoc = fs.readFileSync(config.skillDocPath, 'utf-8');- const personaPrompt = loadPersona(config.persona);-- // Select prompts- let prompts: TestPrompt[];- if (config.holdout && testSet.holdout) {- prompts = testSet.holdout;- } else if (config.sample && config.sample < testSet.prompts.length) {- prompts = stratifiedSample(testSet.prompts, config.sample);- } else {- prompts = testSet.prompts;- }-- // Setup results directory- const resultsDir = path.join(getResultsDir(), iterationId);- ensureDir(resultsDir);- const runsPath = path.join(resultsDir, 'runs.jsonl');-- const results: TestResult[] = [];- const promptsRun: string[] = [];-- console.log(`\nRunning vibeability tests...`);- console.log(` Protocol: ${config.protocol}`);- console.log(` Persona: ${config.persona}`);- console.log(` Model: ${config.model}`);- console.log(- ` Test set: ${testSet.name} (${prompts.length}${config.sample ? ` sampled from ${testSet.prompts.length}` : ''} prompts)`,- );- console.log(` Iteration: ${iterationId}\n`);-- for (let i = 0; i < prompts.length; i++) {- const testPrompt = prompts[i];- promptsRun.push(testPrompt.id);-- process.stdout.write(` ${progressBar(i + 1, prompts.length)}\r`);-- try {- if (config.protocol === 'one-shot') {- const {response, conversation, tokenUsage} = await runOneShot(- client,- testPrompt,- skillDoc,- personaPrompt,- config,- );-- const evaluation = await evaluate(- client,- {- userPrompt: testPrompt.prompt,- llmResponse: response,- expectedComponents: testPrompt.expectedComponents,- skillDoc,- },- config.model,- );-- conversation[0].evaluation = evaluation;-- const result: TestResult = {- id: `${iterationId}-${testPrompt.id}`,- timestamp: timestamp(),- systemVersion: hashContent(skillDoc),- model: config.model,- persona: config.persona,- promptCategory: testPrompt.category,- trajectoryDepth: 0,- prompt: testPrompt.prompt,- response,- evaluation,- fullConversation: conversation,- contextWindowUsage: tokenUsage,- };-- results.push(result);- appendJsonl(runsPath, result);- } else if (config.protocol === 'degradation') {- const {turns, evaluations, tokenUsage} = await runDegradation(- client,- testPrompt,- skillDoc,- personaPrompt,- testSet.fillerPrompts ?? [],- testSet.distractorPrompts ?? [],- testSet.recoveryContext ?? '',- config,- );-- // Create results for each evaluation point- for (const [depth, evaluation] of evaluations) {- const result: TestResult = {- id: `${iterationId}-${testPrompt.id}-t${depth}`,- timestamp: timestamp(),- systemVersion: hashContent(skillDoc),- model: config.model,- persona: config.persona,- promptCategory: testPrompt.category,- trajectoryDepth: depth,- prompt: testPrompt.prompt,- response: turns[depth]?.response ?? '',- evaluation,- fullConversation: turns,- contextWindowUsage: tokenUsage,- };-- results.push(result);- appendJsonl(runsPath, result);- }- }- } catch (error) {- console.error(`\nError on prompt ${testPrompt.id}:`, error);- }-- await sleep(RATE_LIMIT_DELAY);- }-- console.log(`\n`);-- return {- iterationId,- results,- promptsRun,- promptsTotal: testSet.prompts.length,- };-}diff --git a/internal/vibe-tests/src/screenshot-previews.ts b/internal/vibe-tests/src/screenshot-previews.tsnew file mode 100644index 0000000000..3074110612--- /dev/null+++ b/internal/vibe-tests/src/screenshot-previews.ts@@ -0,0 +1,283 @@+#!/usr/bin/env node+/**+ * @file Screenshot pre-built preview HTML files using Playwright+ * @input Pre-built preview HTML files from build-previews.ts+ * @output PNG screenshots at multiple viewport/theme combos+ * @position internal/vibe-tests/src/screenshot-previews.ts+ *+ * Unlike screenshot.ts (which runs a Vite dev server), this script+ * works with pre-built self-contained HTML files. Designed to run in+ * GitHub Actions where Playwright is installed but XDS source isn't needed+ * for rendering.+ *+ * Usage:+ * tsx src/screenshot-previews.ts --iterations 8734233a,d4ff8c2c+ * tsx src/screenshot-previews.ts --iterations 8734233a --prompts tc-4,tc-7+ */++import * as fs from 'node:fs';+import * as path from 'node:path';+import * as http from 'node:http';+import {getResultsDir, ensureDir, writeJson} from './utils.js';++const VIEWPORTS = {+ desktop: {width: 1280, height: 800},+ mobile: {width: 375, height: 812},+};++const THEMES = ['light', 'dark'] as const;++function parseArgs(): {iterations: string[]; prompts?: string[]} {+ const args = process.argv.slice(2);+ let iterations: string[] = [];+ let prompts: string[] | undefined;++ for (let i = 0; i < args.length; i++) {+ if (args[i] === '--iterations' && args[i + 1]) {+ iterations = args[++i].split(',');+ } else if (args[i] === '--prompts' && args[i + 1]) {+ prompts = args[++i].split(',');+ }+ }++ if (iterations.length === 0) {+ console.error(+ 'Usage: tsx src/screenshot-previews.ts --iterations <id1,id2,...> [--prompts <p1,p2,...>]',+ );+ process.exit(1);+ }++ return {iterations, prompts};+}++/**+ * Serve a directory of static files on a random port.+ * Returns the base URL and a close function.+ */+function serveStatic(+ dir: string,+): Promise<{url: string; close: () => Promise<void>}> {+ return new Promise((resolve, reject) => {+ const server = http.createServer((req, res) => {+ const urlPath = decodeURIComponent(req.url || '/');+ const filePath = path.join(dir, urlPath);++ if (!fs.existsSync(filePath)) {+ res.writeHead(404);+ res.end('Not found');+ return;+ }++ const ext = path.extname(filePath).toLowerCase();+ const contentTypes: Record<string, string> = {+ '.html': 'text/html',+ '.css': 'text/css',+ '.js': 'application/javascript',+ '.png': 'image/png',+ '.svg': 'image/svg+xml',+ '.json': 'application/json',+ };++ const content = fs.readFileSync(filePath);+ res.writeHead(200, {'Content-Type': contentTypes[ext] || 'text/plain'});+ res.end(content);+ });++ server.listen(0, '127.0.0.1', () => {+ const addr = server.address();+ if (!addr || typeof addr === 'string') {+ reject(new Error('Failed to get server address'));+ return;+ }+ resolve({+ url: `http://127.0.0.1:${addr.port}`,+ close: () =>+ new Promise<void>(r => {+ server.close(() => r());+ }),+ });+ });++ server.on('error', reject);+ });+}++/**+ * Apply a color-scheme to the page for dark mode screenshots.+ * For self-contained HTML previews, we inject prefers-color-scheme+ * emulation via Playwright's CDP.+ */+async function setTheme(+ page: import('playwright').Page,+ theme: 'light' | 'dark',+) {+ await page.emulateMedia({colorScheme: theme});+}++async function main() {+ const {iterations, prompts} = parseArgs();+ const resultsDir = getResultsDir();++ // Import playwright+ let chromium: typeof import('playwright').chromium;+ try {+ const pw = await import('playwright');+ chromium = pw.chromium;+ } catch {+ console.error(+ 'Playwright is not installed. Run:\n' ++ ' npx playwright install chromium --with-deps\n',+ );+ process.exit(1);+ }++ const browser = await chromium.launch();+ let totalScreenshots = 0;++ try {+ for (const iterationId of iterations) {+ const iterDir = path.join(resultsDir, iterationId);+ const previewsDir = path.join(iterDir, 'previews');+ const screenshotsDir = path.join(iterDir, 'screenshots');++ if (!fs.existsSync(previewsDir)) {+ console.error(+ ` ⚠ No previews directory for ${iterationId}, skipping`,+ );+ continue;+ }++ // Read the preview manifest to find all HTML files+ const manifestPath = path.join(previewsDir, 'manifest.json');+ const previewFiles: Array<{promptId: string; target: string; path: string}> =+ [];++ if (fs.existsSync(manifestPath)) {+ const manifest = JSON.parse(fs.readFileSync(manifestPath, 'utf-8'));+ for (const [promptId, targets] of Object.entries(manifest)) {+ if (prompts && !prompts.includes(promptId)) continue;+ for (const [target, relPath] of Object.entries(+ targets as Record<string, string>,+ )) {+ const fullPath = path.join(iterDir, relPath);+ if (fs.existsSync(fullPath)) {+ previewFiles.push({promptId, target, path: fullPath});+ }+ }+ }+ } else {+ // No manifest — scan for HTML files directly+ const scanDir = (dir: string) => {+ if (!fs.existsSync(dir)) return;+ for (const entry of fs.readdirSync(dir, {withFileTypes: true})) {+ if (entry.isDirectory()) {+ scanDir(path.join(dir, entry.name));+ } else if (entry.name.endsWith('.html')) {+ const promptId = path.basename(path.dirname(path.join(dir, entry.name)));+ const target = path.basename(entry.name, '.html');+ previewFiles.push({+ promptId,+ target,+ path: path.join(dir, entry.name),+ });+ }+ }+ };+ scanDir(previewsDir);+ }++ if (previewFiles.length === 0) {+ console.error(` ⚠ No preview HTML files found for ${iterationId}`);+ continue;+ }++ // Serve the previews directory+ const server = await serveStatic(path.dirname(previewsDir));+ ensureDir(screenshotsDir);++ console.log(+ `\n📸 Screenshotting ${previewFiles.length} preview(s) for ${iterationId}...\n`,+ );++ const screenshotManifest: Record<+ string,+ Record<string, Record<string, Record<string, string>>>+ > = {};++ for (const preview of previewFiles) {+ // Compute the URL relative to the served directory+ const relPath = path.relative(+ path.dirname(previewsDir),+ preview.path,+ );++ for (const [viewportName, viewport] of Object.entries(VIEWPORTS)) {+ for (const theme of THEMES) {+ const page = await browser.newPage({viewport});+ await setTheme(page, theme);++ const url = `${server.url}/${relPath}`;+ try {+ await page.goto(url, {+ waitUntil: 'networkidle',+ timeout: 30000,+ });+ } catch {+ // Fallback: wait for load instead of networkidle+ // (some pages have long-lived connections)+ await page.goto(url, {waitUntil: 'load', timeout: 15000});+ }++ // Wait for fonts and rendering+ await page.waitForTimeout(1000);++ const filename = `${preview.promptId}-${preview.target}-${viewportName}-${theme}.png`;+ const screenshotPath = path.join(screenshotsDir, filename);+ await page.screenshot({path: screenshotPath, fullPage: true});+ await page.close();++ // Track in manifest+ if (!screenshotManifest[preview.promptId]) {+ screenshotManifest[preview.promptId] = {};+ }+ if (!screenshotManifest[preview.promptId][preview.target]) {+ screenshotManifest[preview.promptId][preview.target] = {};+ }+ if (+ !screenshotManifest[preview.promptId][preview.target][+ viewportName+ ]+ ) {+ screenshotManifest[preview.promptId][preview.target][+ viewportName+ ] = {};+ }+ screenshotManifest[preview.promptId][preview.target][viewportName][+ theme+ ] = filename;++ totalScreenshots++;+ console.log(` ✓ ${filename}`);+ }+ }+ }++ // Write screenshot manifest+ writeJson(+ path.join(screenshotsDir, 'manifest.json'),+ screenshotManifest,+ );++ await server.close();+ }+ } finally {+ await browser.close();+ }++ console.log(`\n✅ Captured ${totalScreenshots} screenshot(s)`);+}++main().catch(err => {+ console.error(err);+ process.exit(1);+});