Skip to content

An open source design system that's fully customizable and agent ready

TypeScript

feat(vibe-tests): add GHA screenshot service for nightly pipeline (#853)

Cindy Zhanga1d452a2ac49
diff --git a/.claude/commands/vibe-test.md b/.claude/commands/vibe-test.md
index 6e7a4116df..e6c108e134 100644
--- a/.claude/commands/vibe-test.md
+++ b/.claude/commands/vibe-test.md
@@ -10,7 +10,7 @@ If no argument provided, default to 5 samples.
## Instructions
-Run interactive vibe tests using parallel subagents:
+Run interactive vibe tests using parallel subagents. No API keys required — code generation is done by spawned sub-agents, screenshots by GitHub Actions.
### Step 0: Generate skill doc
@@ -24,122 +24,170 @@ This creates `.generated/xds-skill.md` reflecting all components on the current
### Step 1: Setup the test iteration
-Run the interactive setup command:
+Three targets are compared: *XDS*, *baseline* (shadcn/Tailwind), and *raw HTML/CSS*. All three use the **same prompts** — this is the most important fairness constraint.
```bash
-yarn workspace @xds/vibe-tests interactive --sample <count>
+cd internal/vibe-tests
+
+# 1. Sample prompts for XDS (records the prompt IDs)
+yarn interactive --target xds --persona naive --sample <count>
+# Note the iteration ID and prompt IDs from the output
+
+# 2. Reuse same prompts for baseline
+yarn interactive --target baseline --persona naive --prompts <comma-separated-ids>
+
+# 3. Reuse same prompts for HTML
+yarn interactive --target html --persona naive --prompts <comma-separated-ids>
```
This creates:
+- Three iteration IDs (one per target)
+- Task files in `results/<iteration>/tasks/<promptId>.json`
+- A manifest file per iteration
-- A unique iteration ID
-- Task files in `internal/vibe-tests/results/<iteration>/tasks/`
-- A manifest file
+### Step 2: Spawn sub-agents
-### Step 2: Read the test prompts
+Spawn sub-agents using `spawn_agent` — one per prompt per target (e.g. 10 prompts × 3 targets = 30 agents).
-Read the manifest to get the list of test prompts:
+Each sub-agent must:
+1. **Read the task file** at `results/<iteration>/tasks/<promptId>.json` (on the `xds` node)
+2. **Read the target-specific agent doc:**
+ - XDS → `AGENTS.md` (generated) + run `npx xds component --brief-all` for component catalog
+ - Baseline → `AGENTS.baseline.md` + read `.baseline-docs/`
+ - HTML → `AGENTS.html.md` (no design system docs needed)
+3. **Generate a React component** as a `.tsx` file responding to the prompt
+4. **Self-evaluate** for correctness, accessibility, escape hatches
+5. **Write two files** (on the `xds` node):
+ - `results/<iteration>/results/<promptId>.tsx` — the generated component (must have a default export)
+ - `results/<iteration>/results/<promptId>.json` — metadata:
+ ```json
+ {
+ "docsRead": ["AGENTS.md", "Button.md", "tokens.md"],
+ "completedAt": "2026-03-25T06:00:00Z"
+ }
+ ```
+
+### Step 3: Verify completeness
+
+After all sub-agents complete, verify every task has a `.tsx` and `.json` result:
+
+```bash
+cd internal/vibe-tests
+for ITER in <xds-id> <baseline-id> <html-id>; do
+ EXPECTED=$(ls results/$ITER/tasks/*.json | wc -l)
+ ACTUAL=$(ls results/$ITER/results/*.tsx 2>/dev/null | wc -l)
+ echo "$ITER: $ACTUAL/$EXPECTED complete"
+done
```
-internal/vibe-tests/results/<iteration>/manifest.json
-```
-### Step 3: Run tests in parallel
+Log any gaps. Missing results mean a sub-agent failed — note which prompt/target is affected.
+
+### Step 4: Build previews & screenshots (GitHub Actions)
+
+Commit results to a branch and trigger the screenshot workflow:
-For each test prompt, spawn a subagent using the Task tool with:
+```bash
+cd /vercel/sandbox/repos/xds
+git checkout -b vibe-test-data/<date>
+git add -f internal/vibe-tests/results/
+git commit -m "vibe-test: results for $(date +%Y-%m-%d)"
+git push -u origin vibe-test-data/<date>
+
+# Trigger the screenshot workflow
+gh workflow run vibe-screenshots.yml \
+ --ref vibe-test-data/<date> \
+ -f iterations="<xds-id>,<baseline-id>,<html-id>"
+```
-- `subagent_type`: "general-purpose"
-- Read the generated skill doc at `internal/vibe-tests/.generated/xds-skill.md`
-- Generate code for the prompt using XDS components
-- Self-evaluate the response for:
- - Correct components used
- - No hallucinated props
- - Acceptable vs critical escape hatches
-- Track timing: record start time before generating, calculate durationMs after
-- Return JSON with: response, evaluation, durationMs, inputTokens, outputTokens
+The workflow:
+1. Builds self-contained HTML previews (inlines all CSS via vite-plugin-singlefile)
+2. Captures Playwright screenshots at 4 combos: desktop/mobile × light/dark
+3. Uploads artifacts: `preview-html` and `vibe-test-screenshots`
-### Step 4: Write results
+### Step 5: Wait for GHA and download
-After all subagents complete, write results to:
+Poll the run until complete:
+```bash
+RUN_ID=$(gh run list --workflow=vibe-screenshots.yml --limit 1 --json databaseId --jq '.[0].databaseId')
+gh run watch $RUN_ID
```
-internal/vibe-tests/results/<iteration>/runs.jsonl
+
+Download artifacts:
+
+```bash
+gh run download $RUN_ID -n vibe-test-screenshots -D internal/vibe-tests/results/
+gh run download $RUN_ID -n preview-html -D internal/vibe-tests/results/
```
-Each line should be a JSON object with:
-
-```json
-{
- "id": "<iteration>-<promptId>",
- "timestamp": "<ISO timestamp>",
- "systemVersion": "xds-llms-v1",
- "model": "claude-code-interactive",
- "persona": "naive",
- "promptCategory": "<category>",
- "trajectoryDepth": 0,
- "prompt": "<the prompt>",
- "response": "<generated code>",
- "evaluation": {
- "success": true/false,
- "componentsUsed": ["XDSCard", ...],
- "componentsExpected": ["XDSCard", ...],
- "escapeHatches": [{"type": "...", "severity": "acceptable/critical", "detail": "...", "codeSnippet": "..."}],
- "failureMode": null or "description",
- "confusionSignals": []
- },
- "fullConversation": [],
- "contextWindowUsage": 0,
- "durationMs": <milliseconds taken>,
- "inputTokens": <input tokens used>,
- "outputTokens": <output tokens generated>
-}
+If GHA fails: `gh run view $RUN_ID --log-failed`
+
+### Step 6: Aggregate and report
+
+```bash
+cd internal/vibe-tests
+yarn aggregate --iteration <xds-id>
+yarn report:deploy --iteration <xds-id> --baseline <baseline-id> --html <html-id>
```
-### Step 5: Aggregate results
+Report URL: `https://facebookexperimental.github.io/xds/reports/<xds-id>/`
+
+### Step 7: Post results
-Run:
+Create a GitHub issue:
```bash
-yarn workspace @xds/vibe-tests aggregate --iteration <id>
+gh issue create --repo facebookexperimental/xds \
+ --title "Vibe Test Results — $(date +%Y-%m-%d)" \
+ --label "vibe-test" \
+ --body "<results summary>"
```
-This will print a summary showing:
+Include: overall scores, dimension winners, report link, gaps, and screenshots.
-- Overall success rate
-- Performance stats (total time, avg time, tokens used)
-- Success by category
-- Critical issues (if any)
-- Acceptable escape hatches
+## Fairness Rules
-It also generates an HTML report at `results/<iteration>/report.html` that you can open in a browser to inspect individual test results.
+These rules ensure the comparison is meaningful:
-### Evaluation Criteria
+1. **Same prompts** — all targets run the exact same prompt IDs (use `--prompts`)
+2. **Same persona** — always `naive` (simulates a developer with no prior knowledge)
+3. **Fair docs** — each target gets its own equivalent agent doc, not the other target's
+4. **No cross-contamination** — XDS agents don't see baseline docs, and vice versa
+5. **10 prompts minimum** — stratified sample covers all categories
-**Quality Tiers:**
+## Evaluation Criteria
-- 🥇 **Gold** - Pure XDS, no escape hatches needed
-- 🟢 **Green** - Correct components, only acceptable escape hatches
-- 🟡 **Yellow** - Anti-patterns that break theming (hardcoded values)
-- 🔴 **Red** - Critical failures
+### Quality Tiers
-**Critical escape hatches (Red tier):**
+- 🥇 **Gold** — Pure target usage, no escape hatches needed
+- 🟢 **Green** — Correct components, only acceptable escape hatches
+- 🟡 **Yellow** — Anti-patterns that break theming (hardcoded values)
+- 🔴 **Red** — Critical failures (hallucinations, wrong components)
-- `hallucination` - Invented props/components that don't exist
-- `wrong_component` - Using a component incorrectly
-- `redundant_css` - CSS that duplicates a component prop
+### Escape Hatch Classification
-**Anti-pattern escape hatches (Yellow tier - break theming):**
+**Critical (Red tier):**
+- `hallucination` — invented props/components that don't exist
+- `wrong_component` — using a component incorrectly
+- `redundant_css` — CSS that duplicates a component prop
-- `hardcoded_color` - Direct color values instead of CSS variables
-- `hardcoded_spacing` - Direct pixel values for spacing instead of CSS variables
-- `hardcoded_size` - Direct pixel values for sizes instead of CSS variables
-- `inline_style` - Inline styles instead of StyleX
+**Anti-pattern (Yellow tier):**
+- `hardcoded_color` — direct color values instead of CSS variables
+- `hardcoded_spacing` — direct pixel values instead of CSS variables
+- `inline_style` — inline styles instead of StyleX
-**Acceptable escape hatches (Green tier):**
+**Acceptable (Green tier):**
+- `supplemental_css` — StyleX for things the system doesn't cover
+- `wrapper_div` — structural HTML for composition
+- `custom_animation` — animation not covered by the system
-- `supplemental_css` - StyleX for things XDS doesn't cover
-- `wrapper_div` - Structural HTML for composition
-- `custom_animation` - Animation not covered by XDS
-- `layout_workaround` - Layout patterns XDS doesn't support
+## Evaluation Dimensions
-When evaluating, also include a `gap` field on escape hatches describing what capability is missing, to help generate improvement suggestions.
+| Dimension | What It Measures |
+|-----------|-----------------|
+| Correctness | Valid component usage, no hallucinations |
+| Accessibility | Labels, semantics, keyboard support |
+| Code Quality | Complexity, patterns, TypeScript usage |
+| Efficiency | Decisions/element, DRYness, conciseness |
+| Maintainability | Semantic tokens vs magic values, dark mode, locality |
diff --git a/.github/workflows/vibe-screenshots.yml b/.github/workflows/vibe-screenshots.yml
new file mode 100644
index 0000000000..59e03665c9
--- /dev/null
+++ b/.github/workflows/vibe-screenshots.yml
@@ -0,0 +1,100 @@
+name: Vibe Test Screenshots
+
+on:
+ workflow_dispatch:
+ inputs:
+ iterations:
+ description: 'Comma-separated iteration IDs to screenshot'
+ required: true
+ prompts:
+ description: 'Comma-separated prompt IDs (optional, defaults to all)'
+ required: false
+ default: ''
+
+concurrency:
+ group: vibe-screenshots-${{ github.run_id }}
+ cancel-in-progress: false
+
+jobs:
+ build-previews:
+ name: Build Previews
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout
+ uses: actions/checkout@v4
+
+ - name: Setup Node.js
+ uses: actions/setup-node@v4
+ with:
+ node-version: '22'
+ cache: 'yarn'
+
+ - name: Install dependencies
+ run: yarn install --frozen-lockfile
+
+ - name: Build core packages
+ run: yarn build
+
+ - name: Build preview HTML files
+ run: |
+ cd internal/vibe-tests
+ PROMPT_ARG=""
+ if [ -n "${{ github.event.inputs.prompts }}" ]; then
+ PROMPT_ARG="--prompts ${{ github.event.inputs.prompts }}"
+ fi
+ npx tsx src/build-previews.ts \
+ --iterations "${{ github.event.inputs.iterations }}" \
+ $PROMPT_ARG
+
+ - name: Upload preview HTML
+ uses: actions/upload-artifact@v4
+ with:
+ name: preview-html
+ path: internal/vibe-tests/results/*/previews/
+ retention-days: 90
+
+ screenshot:
+ name: Capture Screenshots
+ needs: build-previews
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout
+ uses: actions/checkout@v4
+
+ - name: Setup Node.js
+ uses: actions/setup-node@v4
+ with:
+ node-version: '22'
+ cache: 'yarn'
+
+ - name: Install dependencies
+ run: yarn install --frozen-lockfile
+
+ - name: Install Playwright
+ run: npx playwright install chromium --with-deps
+
+ - name: Download preview HTML
+ uses: actions/download-artifact@v4
+ with:
+ name: preview-html
+ path: internal/vibe-tests/results/
+
+ - name: Screenshot all previews
+ run: |
+ cd internal/vibe-tests
+ npx tsx src/screenshot-previews.ts \
+ --iterations "${{ github.event.inputs.iterations }}"
+
+ - name: Upload screenshots
+ uses: actions/upload-artifact@v4
+ with:
+ name: vibe-test-screenshots
+ path: internal/vibe-tests/results/*/screenshots/
+ retention-days: 90
+
+ - name: Upload screenshot manifest
+ uses: actions/upload-artifact@v4
+ with:
+ name: screenshot-manifest
+ path: internal/vibe-tests/results/*/screenshots/manifest.json
+ retention-days: 90
diff --git a/.github/workflows/vibe-tests.yml.disabled b/.github/workflows/vibe-tests.yml.disabled
deleted file mode 100644
index f98ccc0943..0000000000
--- a/.github/workflows/vibe-tests.yml.disabled
+++ /dev/null
@@ -1,224 +0,0 @@
-# =============================================================================
-# DISABLED: This workflow is temporarily disabled.
-#
-# To re-enable:
-# 1. Add ANTHROPIC_API_KEY secret to the repository
-# 2. Rename this file back to vibe-tests.yml
-# =============================================================================
-
-name: Vibe Tests
-
-on:
- pull_request:
- branches: [main, master]
- push:
- branches: [main, master]
- workflow_dispatch:
- inputs:
- sample_size:
- description: 'Number of test prompts to run'
- required: false
- default: '10'
-
-concurrency:
- group: vibe-tests-${{ github.head_ref || github.ref }}
- cancel-in-progress: true
-
-jobs:
- vibe-test:
- name: Run Vibe Tests
- runs-on: ubuntu-latest
- permissions:
- contents: read
- pull-requests: write
- actions: read
-
- steps:
- - name: Checkout
- uses: actions/checkout@v4
-
- - name: Setup Node.js
- uses: actions/setup-node@v4
- with:
- node-version: '22'
- cache: 'yarn'
-
- - name: Install dependencies
- run: yarn install --frozen-lockfile
-
- - name: Run vibe tests
- id: vibe-test
- env:
- ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
- SKILL_DOC: packages/core/xds.md
- run: |
- # Determine sample size
- SAMPLE_SIZE="${{ github.event.inputs.sample_size || '10' }}"
-
- # Run the tests
- echo "Running vibe tests with sample size: $SAMPLE_SIZE"
- yarn workspace @xds/vibe-tests test:xds:sample 2>&1 | tee vibe-test-output.txt
-
- # Extract iteration ID from output
- ITERATION_ID=$(grep -oP 'Iteration: \K[a-f0-9]+' vibe-test-output.txt | head -1)
- echo "iteration_id=$ITERATION_ID" >> $GITHUB_OUTPUT
- echo "Iteration ID: $ITERATION_ID"
-
- - name: Aggregate results
- id: aggregate
- if: steps.vibe-test.outputs.iteration_id != ''
- run: |
- ITERATION_ID="${{ steps.vibe-test.outputs.iteration_id }}"
- yarn workspace @xds/vibe-tests aggregate --iteration "$ITERATION_ID" --ci
-
- - name: Upload results artifact
- if: steps.vibe-test.outputs.iteration_id != ''
- uses: actions/upload-artifact@v4
- with:
- name: vibe-test-results-${{ steps.vibe-test.outputs.iteration_id }}
- path: |
- internal/vibe-tests/results/${{ steps.vibe-test.outputs.iteration_id }}/
- retention-days: 90
-
- - name: Upload HTML report
- if: steps.vibe-test.outputs.iteration_id != ''
- uses: actions/upload-artifact@v4
- with:
- name: vibe-test-report-${{ steps.vibe-test.outputs.iteration_id }}
- path: internal/vibe-tests/results/${{ steps.vibe-test.outputs.iteration_id }}/report.html
- retention-days: 90
-
- - name: Comment on PR
- if: github.event_name == 'pull_request' && steps.vibe-test.outputs.iteration_id != ''
- uses: actions/github-script@v7
- with:
- script: |
- const successRate = '${{ steps.aggregate.outputs.success_rate }}';
- const gold = '${{ steps.aggregate.outputs.gold }}' || '0';
- const green = '${{ steps.aggregate.outputs.green }}' || '0';
- const yellow = '${{ steps.aggregate.outputs.yellow }}' || '0';
- const red = '${{ steps.aggregate.outputs.red }}' || '0';
- const iterationId = '${{ steps.vibe-test.outputs.iteration_id }}';
- const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
-
- // Determine status emoji
- let statusEmoji = '🔴';
- if (parseInt(successRate) >= 90) statusEmoji = '🟢';
- else if (parseInt(successRate) >= 70) statusEmoji = '🟡';
-
- const body = `## ${statusEmoji} Vibe Test Results
-
- **Iteration:** \`${iterationId}\`
- **Success Rate:** ${successRate}%
-
- ### Quality Tiers
- | Tier | Count |
- |------|-------|
- | 🥇 Gold (pure XDS) | ${gold} |
- | 🟢 Green (acceptable) | ${green} |
- | 🟡 Yellow (anti-pattern) | ${yellow} |
- | 🔴 Red (critical) | ${red} |
-
- <details>
- <summary>Tier Definitions</summary>
-
- - **Gold**: Pure XDS, no escape hatches needed
- - **Green**: Correct components with acceptable escape hatches
- - **Yellow**: Anti-patterns that break theming (hardcoded values, inline styles)
- - **Red**: Critical failures (hallucinations, wrong components)
- </details>
-
- 📊 [View full report](${runUrl}) (download artifacts)
- `;
-
- // Find existing comment
- const { data: comments } = await github.rest.issues.listComments({
- owner: context.repo.owner,
- repo: context.repo.repo,
- issue_number: context.issue.number,
- });
-
- const botComment = comments.find(comment =>
- comment.user.type === 'Bot' &&
- comment.body.includes('Vibe Test Results')
- );
-
- if (botComment) {
- await github.rest.issues.updateComment({
- owner: context.repo.owner,
- repo: context.repo.repo,
- comment_id: botComment.id,
- body: body
- });
- } else {
- await github.rest.issues.createComment({
- owner: context.repo.owner,
- repo: context.repo.repo,
- issue_number: context.issue.number,
- body: body
- });
- }
-
- - name: Check threshold
- if: steps.aggregate.outputs.success_rate != ''
- run: |
- SUCCESS_RATE="${{ steps.aggregate.outputs.success_rate }}"
- RED="${{ steps.aggregate.outputs.red }}"
-
- echo "✅ Success rate: $SUCCESS_RATE%"
- echo "✅ Critical failures: $RED"
-
- # Fail if success rate is below 70%
- if [ "$SUCCESS_RATE" -lt 70 ]; then
- echo "::error::Vibe test success rate ($SUCCESS_RATE%) is below threshold (70%)"
- exit 1
- fi
-
- echo "✅ Vibe tests passed with $SUCCESS_RATE% success rate"
-
- store-results:
- name: Store Historical Results
- needs: vibe-test
- if: github.event_name == 'push' && github.ref == 'refs/heads/main'
- runs-on: ubuntu-latest
- permissions:
- contents: write
-
- steps:
- - name: Checkout
- uses: actions/checkout@v4
- with:
- ref: vibe-test-history
- fetch-depth: 0
-
- - name: Download artifacts
- uses: actions/download-artifact@v4
- with:
- pattern: vibe-test-results-*
- path: results/
- merge-multiple: true
-
- - name: Commit results
- run: |
- git config user.name "github-actions[bot]"
- git config user.email "github-actions[bot]@users.noreply.github.com"
-
- # Create branch if it doesn't exist
- if ! git rev-parse --verify vibe-test-history >/dev/null 2>&1; then
- git checkout --orphan vibe-test-history
- git rm -rf . || true
- echo "# Vibe Test Results History" > README.md
- echo "" >> README.md
- echo "This branch stores historical vibe test results from CI runs." >> README.md
- fi
-
- # Add new results
- TIMESTAMP=$(date +%Y-%m-%d-%H%M%S)
- COMMIT_SHA="${{ github.sha }}"
- mkdir -p "history/${TIMESTAMP}-${COMMIT_SHA:0:7}"
- cp -r results/* "history/${TIMESTAMP}-${COMMIT_SHA:0:7}/" || true
-
- # Commit and push
- git add .
- git commit -m "Add vibe test results from ${COMMIT_SHA:0:7}" || echo "No changes to commit"
- git push origin vibe-test-history || echo "Failed to push, branch may not exist"
diff --git a/CLAUDE.md b/CLAUDE.md
index b181ba85b3..ca5d794d61 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -18,13 +18,15 @@ Tests how well AGENTS.md helps LLMs generate correct XDS component code.
**How to execute:**
-1. Run `yarn workspace @xds/vibe-tests interactive --sample <count> [--degradation]` to set up iteration
-2. Spawn parallel subagents (one per test prompt) with `mode: "bypassPermissions"` to:
+1. Run `yarn workspace @xds/vibe-tests interactive --sample <count>` to set up iteration
+2. Spawn parallel subagents (one per test prompt) to:
- Read the task file from `results/<iteration>/tasks/{promptId}.json`
- Generate code for the prompt using XDS components (AGENTS.md auto-injected)
- Self-evaluate for success/escape hatches
- - Write result to individual file: `results/<iteration>/results/{promptId}.json`
-3. Run `yarn workspace @xds/vibe-tests aggregate --iteration <id>` to see results
+ - Write `.tsx` result to `results/<iteration>/results/{promptId}.tsx`
+ - Write `.json` metadata to `results/<iteration>/results/{promptId}.json`
+3. Trigger `gh workflow run vibe-screenshots.yml` to build previews and capture screenshots
+4. Run `yarn workspace @xds/vibe-tests aggregate --iteration <id>` to see results
**Degradation mode (--degradation):**
Tests context retention across 10-turn conversations with filler, distractor, and recovery turns.
diff --git a/internal/vibe-tests/README.md b/internal/vibe-tests/README.md
index 49baa3bbce..11664a680a 100644
--- a/internal/vibe-tests/README.md
+++ b/internal/vibe-tests/README.md
@@ -41,8 +41,13 @@ The skill doc is generated fresh each time, so it always reflects the current br
## Quick Start - API Mode
```bash
-# Requires ANTHROPIC_API_KEY
+# Requires ANTHROPIC_API_KEY (direct Anthropic mode)
yarn workspace @xds/vibe-tests test:xds:sample
+
+# Or use the Navi-orchestrated pipeline (no API key needed):
+# 1. Navi sub-agents generate TSX results
+# 2. GHA builds previews and captures screenshots
+# See: .github/workflows/vibe-screenshots.yml
```
## What It Does
@@ -234,29 +239,25 @@ Suggested refinements:
## CI Integration
-Vibe tests run automatically on every PR via GitHub Actions.
+### Screenshot Service (`.github/workflows/vibe-screenshots.yml`)
-### GitHub Action
+The primary workflow for nightly vibe tests. Triggered by Navi via `workflow_dispatch`:
-The workflow (`.github/workflows/vibe-tests.yml`) runs on:
+```bash
+gh workflow run vibe-screenshots.yml -f iterations="<id1>,<id2>,<id3>"
+```
-- Every pull request to main/master
-- Every push to main/master
-- Manual trigger with configurable sample size
+**What it does:**
-### What it does:
+1. Builds self-contained HTML previews from TSX results (inlines all CSS via vite-plugin-singlefile)
+2. Captures Playwright screenshots at 4 combos: desktop/mobile × light/dark
+3. Uploads preview HTML and screenshots as artifacts (90-day retention)
-1. Runs 10 stratified sample tests
-2. Aggregates results and generates HTML report
-3. Posts a summary comment on the PR
-4. Uploads results as artifacts (90-day retention)
-5. Fails the check if:
- - Success rate is below 70%
- - Any critical (Red tier) failures
+**No API keys required.** Code generation is handled by Navi sub-agents before this workflow runs.
-### Required Secrets:
+### Direct API Mode (legacy)
-- `ANTHROPIC_API_KEY`: API key for running tests
+The `test:xds:sample` script calls Anthropic directly and requires `ANTHROPIC_API_KEY`. This is the old all-in-one flow — prefer the Navi-orchestrated pipeline for nightly runs.
### Aggregate CLI Flags:
diff --git a/internal/vibe-tests/package.json b/internal/vibe-tests/package.json
index 6a49b47ee7..a2a8951a78 100644
--- a/internal/vibe-tests/package.json
+++ b/internal/vibe-tests/package.json
@@ -10,10 +10,6 @@
"scripts": {
"harness": "tsx src/cli.ts",
"generate:skill-doc": "bash scripts/generate-skill-doc.sh",
- "test:xds": "yarn generate:skill-doc && SKILL_DOC=.generated/xds-skill.md tsx src/cli.ts run",
- "test:xds:sample": "yarn generate:skill-doc && SKILL_DOC=.generated/xds-skill.md tsx src/cli.ts run --sample 10",
- "test:xds:holdout": "yarn generate:skill-doc && SKILL_DOC=.generated/xds-skill.md tsx src/cli.ts run --holdout",
- "test:ci": "yarn generate:skill-doc && SKILL_DOC=.generated/xds-skill.md tsx src/cli.ts run --sample 10",
"interactive": "tsx src/interactive.ts",
"interactive:sample": "tsx src/interactive.ts --sample 5",
"run-interactive": "tsx src/run-interactive.ts",
@@ -21,7 +17,6 @@
"aggregate": "tsx src/aggregate.ts",
"aggregate:ci": "tsx src/aggregate.ts --ci",
"aggregate:json": "tsx src/aggregate.ts --json",
- "analyze": "tsx src/cli.ts analyze",
"quality": "tsx src/quality.ts",
"compare": "tsx src/compare.ts",
"generate-a11y-manifest": "tsx src/generate-a11y-manifest.ts",
@@ -31,10 +26,10 @@
"report:dev": "tsx src/build-report.ts --dev",
"report:deploy": "tsx src/deploy-report.ts",
"preview:dev": "vite --config app/vite.config.preview.ts",
- "preview:build": "vite build --config app/vite.config.preview.ts"
+ "preview:build": "vite build --config app/vite.config.preview.ts",
+ "screenshot:previews": "tsx src/screenshot-previews.ts"
},
"dependencies": {
- "@anthropic-ai/sdk": "^0.52.0",
"@xds/cli": "*",
"commander": "^12.1.0"
},
diff --git a/internal/vibe-tests/src/analyst.ts b/internal/vibe-tests/src/analyst.ts
deleted file mode 100644
index 4a0c30e7f5..0000000000
--- a/internal/vibe-tests/src/analyst.ts
+++ /dev/null
@@ -1,101 +0,0 @@
-/**
- * @file Analyst module - analyzes test results and suggests refinements
- * @position internal/vibe-tests/src/analyst.ts
- */
-
-import Anthropic from '@anthropic-ai/sdk';
-import * as path from 'node:path';
-import type {TestResult, AnalysisResult} from './types.js';
-import {readPromptFile, getPromptsDir} from './utils.js';
-
-const ANALYST_PROMPT_PATH = path.join(getPromptsDir(), 'analyst.md');
-
-export interface AnalyzeInput {
- results: TestResult[];
- skillDoc: string;
- promptsRun: number;
- promptsTotal: number;
-}
-
-/**
- * Analyze a batch of test results and suggest refinements
- */
-export async function analyze(
- client: Anthropic,
- input: AnalyzeInput,
- model: string,
-): Promise<AnalysisResult> {
- const analystPrompt = readPromptFile(ANALYST_PROMPT_PATH);
-
- const systemPrompt = `${analystPrompt}
-
-## Skill Doc (Component System Documentation)
-
-${input.skillDoc}`;
-
- // Prepare results summary (full results could be too large)
- const resultsSummary = input.results.map(r => ({
- id: r.id,
- category: r.promptCategory,
- persona: r.persona,
- prompt: r.prompt,
- success: r.evaluation.success,
- componentsUsed: r.evaluation.componentsUsed,
- componentsExpected: r.evaluation.componentsExpected,
- escapeHatches: r.evaluation.escapeHatches,
- failureMode: r.evaluation.failureMode,
- confusionSignals: r.evaluation.confusionSignals,
- trajectoryDepth: r.trajectoryDepth,
- }));
-
- const userMessage = `## Sampling Metadata
-
-- Prompts run: ${input.promptsRun}
-- Total prompts in pool: ${input.promptsTotal}
-- Sampling: ${input.promptsRun < input.promptsTotal ? 'Yes (stratified)' : 'No (full run)'}
-
-## Test Results
-
-\`\`\`json
-${JSON.stringify(resultsSummary, null, 2)}
-\`\`\`
-
----
-
-Analyze these results and return JSON matching the AnalysisResult schema. Only output valid JSON, no other text.`;
-
- const response = await client.messages.create({
- model,
- max_tokens: 4096,
- system: systemPrompt,
- messages: [{role: 'user', content: userMessage}],
- });
-
- const content = response.content[0];
- if (content.type !== 'text') {
- throw new Error('Expected text response from analyst');
- }
-
- // Parse JSON from response
- let jsonText = content.text.trim();
- if (jsonText.startsWith('```json')) {
- jsonText = jsonText.slice(7);
- } else if (jsonText.startsWith('```')) {
- jsonText = jsonText.slice(3);
- }
- if (jsonText.endsWith('```')) {
- jsonText = jsonText.slice(0, -3);
- }
- jsonText = jsonText.trim();
-
- try {
- return JSON.parse(jsonText) as AnalysisResult;
- } catch {
- console.error('Failed to parse analyst response:', content.text);
- return {
- patterns: [],
- refinements: [],
- summary: 'Failed to parse analyst response',
- };
- }
-}
diff --git a/internal/vibe-tests/src/cli.ts b/internal/vibe-tests/src/cli.ts
index d45d1c8adb..9f94e7a854 100755
--- a/internal/vibe-tests/src/cli.ts
+++ b/internal/vibe-tests/src/cli.ts
@@ -2,19 +2,18 @@
/**
* @file CLI entry point for vibe-tests harness
* @position internal/vibe-tests/src/cli.ts
+ *
+ * Code generation is handled by Navi sub-agents (no API key needed).
+ * This CLI provides aggregation, reporting, and utility commands.
*/
import {Command} from 'commander';
import * as path from 'node:path';
import * as fs from 'node:fs';
-import Anthropic from '@anthropic-ai/sdk';
-import type {RunConfig, PersonaType, Protocol, Iteration} from './types.js';
-import {runTests} from './runner.js';
-import {analyze} from './analyst.js';
+import type {Iteration} from './types.js';
import {
loadTestSet,
readJsonl,
- writeJson,
readJson,
getResultsDir,
} from './utils.js';
@@ -27,162 +26,6 @@ program
.description('Vibeability test harness for LLM component system usage')
.version('0.0.1');
-/**
- * Run command
- */
-program
- .command('run')
- .description('Run vibeability tests')
- .option('--protocol <type>', 'Test protocol', 'one-shot')
- .option('--persona <type>', 'User persona', 'naive')
- .option('--model <model>', 'LLM model', 'claude-sonnet-4-20250514')
- .option('--skill-doc <path>', 'Path to skill doc')
- .option('--sample <n>', 'Sample N prompts (stratified)', parseInt)
- .option('--holdout', 'Run holdout set instead of main prompts')
- .action(async options => {
- const skillDocPath =
- options.skillDoc ?? process.env.SKILL_DOC ?? './skills/brownie.md';
-
- if (!fs.existsSync(skillDocPath)) {
- console.error(`Error: Skill doc not found at ${skillDocPath}`);
- console.error(
- 'Set SKILL_DOC env var or use --skill-doc flag to specify path',
- );
- process.exit(1);
- }
-
- const config: RunConfig = {
- protocol: options.protocol as Protocol,
- persona: options.persona as PersonaType,
- model: options.model,
- skillDocPath,
- sample: options.sample,
- holdout: options.holdout,
- };
-
- const testSet = loadTestSet('default');
- const result = await runTests(testSet, config);
-
- // Print results summary
- console.log('Results:');
- console.log(
- ` Overall: ${calculateSuccessRate(result.results)}% success (${result.results.filter(r => r.evaluation.success).length}/${result.results.length})\n`,
- );
-
- // By category
- console.log(' By category:');
- const byCategory = groupBy(result.results, r => r.promptCategory);
- for (const [category, results] of Object.entries(byCategory)) {
- console.log(` ${category}: ${calculateSuccessRate(results)}%`);
- }
-
- // Escape hatches
- console.log('\n Critical issues:');
- const criticalHatches = result.results.flatMap(r =>
- r.evaluation.escapeHatches.filter(
- (h): h is import('./types.js').EscapeHatch =>
- typeof h !== 'string' && h.severity === 'critical',
- ),
- );
- const criticalCounts = countBy(criticalHatches, h => h.type);
- for (const [type, count] of Object.entries(criticalCounts)) {
- console.log(` - ${type} (${count})`);
- }
-
- console.log('\n Acceptable escape hatches:');
- const acceptableHatches = result.results.flatMap(r =>
- r.evaluation.escapeHatches.filter(
- (h): h is import('./types.js').EscapeHatch =>
- typeof h !== 'string' && h.severity === 'acceptable',
- ),
- );
- const acceptableCounts = countBy(acceptableHatches, h => h.type);
- for (const [type, count] of Object.entries(acceptableCounts)) {
- console.log(` - ${type} (${count})`);
- }
-
- console.log(
- `\n Results saved: ${path.join(getResultsDir(), result.iterationId, 'runs.jsonl')}`,
- );
- console.log(
- `\nRun \`vibe-tests analyze --iteration ${result.iterationId}\` for refinement suggestions.`,
- );
- });
-
-/**
- * Analyze command
- */
-program
- .command('analyze')
- .description('Analyze test results from an iteration')
- .requiredOption('--iteration <id>', 'Iteration ID to analyze')
- .option(
- '--model <model>',
- 'LLM model for analysis',
- 'claude-sonnet-4-20250514',
- )
- .option('--skill-doc <path>', 'Path to skill doc')
- .action(async options => {
- const iterationId = options.iteration;
- const resultsDir = path.join(getResultsDir(), iterationId);
-
- if (!fs.existsSync(resultsDir)) {
- console.error(`Error: Iteration ${iterationId} not found`);
- process.exit(1);
- }
-
- const runsPath = path.join(resultsDir, 'runs.jsonl');
- const results = readJsonl<TestResult>(runsPath);
-
- if (results.length === 0) {
- console.error(`Error: No results found for iteration ${iterationId}`);
- process.exit(1);
- }
-
- const skillDocPath =
- options.skillDoc ??
- process.env.SKILL_DOC ??
- `./skills/${results[0].systemVersion}.md`;
-
- if (!fs.existsSync(skillDocPath)) {
- console.error(`Error: Skill doc not found at ${skillDocPath}`);
- process.exit(1);
- }
-
- const skillDoc = fs.readFileSync(skillDocPath, 'utf-8');
- const promptsRun = new Set(results.map(r => r.id.split('-')[1])).size;
- const promptsTotal = loadTestSet('default').prompts.length;
-
- console.log(`\nAnalyzing iteration ${iterationId}...`);
-
- const client = new Anthropic();
- const analysis = await analyze(
- client,
- {results, skillDoc, promptsRun, promptsTotal},
- options.model,
- );
-
- // Save analysis
- const analysisPath = path.join(resultsDir, 'analysis.json');
- writeJson(analysisPath, analysis);
-
- // Print results
- console.log('\nPatterns detected:');
- for (const pattern of analysis.patterns) {
- console.log(` - ${pattern}`);
- }
-
- console.log('\nSuggested refinements:');
- for (let i = 0; i < analysis.refinements.length; i++) {
- const ref = analysis.refinements[i];
- console.log(
- ` ${i + 1}. [${ref.target}] ${ref.suggestion} (confidence: ${ref.confidence.toFixed(1)}, effort: ${ref.effortEstimate})`,
- );
- }
-
- console.log(`\nSaved: ${analysisPath}`);
- });
-
/**
* Report command
*/
@@ -227,35 +70,4 @@ program
}
});
-// Helper functions
-function calculateSuccessRate(results: TestResult[]): string {
- const successes = results.filter(r => r.evaluation.success).length;
- return ((successes / results.length) * 100).toFixed(0);
-}
-
-function groupBy<T>(
- items: T[],
- keyFn: (item: T) => string,
-): Record<string, T[]> {
- const groups: Record<string, T[]> = {};
- for (const item of items) {
- const key = keyFn(item);
- if (!groups[key]) groups[key] = [];
- groups[key].push(item);
- }
- return groups;
-}
-
-function countBy<T>(
- items: T[],
- keyFn: (item: T) => string,
-): Record<string, number> {
- const counts: Record<string, number> = {};
- for (const item of items) {
- const key = keyFn(item);
- counts[key] = (counts[key] ?? 0) + 1;
- }
- return counts;
-}
-
program.parse();
diff --git a/internal/vibe-tests/src/evaluator.ts b/internal/vibe-tests/src/evaluator.ts
deleted file mode 100644
index 203f942bb8..0000000000
--- a/internal/vibe-tests/src/evaluator.ts
+++ /dev/null
@@ -1,90 +0,0 @@
-/**
- * @file Evaluator module - judges LLM responses against component system
- * @position internal/vibe-tests/src/evaluator.ts
- */
-
-import Anthropic from '@anthropic-ai/sdk';
-import * as path from 'node:path';
-import type {Evaluation} from './types.js';
-import {readPromptFile, getPromptsDir} from './utils.js';
-
-const EVALUATOR_PROMPT_PATH = path.join(getPromptsDir(), 'evaluator.md');
-
-export interface EvaluateInput {
- userPrompt: string;
- llmResponse: string;
- expectedComponents: string[];
- skillDoc: string;
-}
-
-/**
- * Evaluate an LLM response against the component system
- */
-export async function evaluate(
- client: Anthropic,
- input: EvaluateInput,
- model: string,
-): Promise<Evaluation> {
- const evaluatorPrompt = readPromptFile(EVALUATOR_PROMPT_PATH);
-
- const systemPrompt = `${evaluatorPrompt}
-
-## Skill Doc (Component System Documentation)
-
-${input.skillDoc}`;
-
- const userMessage = `## User's Original Prompt
-
-${input.userPrompt}
-
-## Expected Components
-
-${input.expectedComponents.join(', ')}
-
-## LLM Response to Evaluate
-
-${input.llmResponse}
-
----
-
-Evaluate this response and return JSON matching the Evaluation schema. Only output valid JSON, no other text.`;
-
- const response = await client.messages.create({
- model,
- max_tokens: 2048,
- system: systemPrompt,
- messages: [{role: 'user', content: userMessage}],
- });
-
- const content = response.content[0];
- if (content.type !== 'text') {
- throw new Error('Expected text response from evaluator');
- }
-
- // Parse JSON from response, handling potential markdown code blocks
- let jsonText = content.text.trim();
- if (jsonText.startsWith('```json')) {
- jsonText = jsonText.slice(7);
- } else if (jsonText.startsWith('```')) {
- jsonText = jsonText.slice(3);
- }
- if (jsonText.endsWith('```')) {
- jsonText = jsonText.slice(0, -3);
- }
- jsonText = jsonText.trim();
-
- try {
- return JSON.parse(jsonText) as Evaluation;
- } catch {
- // If parsing fails, return a failure evaluation
- console.error('Failed to parse evaluator response:', content.text);
- return {
- success: false,
- componentsUsed: [],
- componentsExpected: input.expectedComponents,
- escapeHatches: [],
- failureMode: 'evaluator_parse_error',
- confusionSignals: [],
- };
- }
-}
diff --git a/internal/vibe-tests/src/index.ts b/internal/vibe-tests/src/index.ts
index b956cf2388..0fca89e461 100644
--- a/internal/vibe-tests/src/index.ts
+++ b/internal/vibe-tests/src/index.ts
@@ -4,7 +4,4 @@
*/
export * from './types.js';
-export {runTests} from './runner.js';
-export {evaluate} from './evaluator.js';
-export {analyze} from './analyst.js';
export {stratifiedSample} from './sampling.js';
diff --git a/internal/vibe-tests/src/runner.ts b/internal/vibe-tests/src/runner.ts
deleted file mode 100644
index 71233b025c..0000000000
--- a/internal/vibe-tests/src/runner.ts
+++ /dev/null
@@ -1,374 +0,0 @@
-/**
- * @file Test runner - orchestrates test execution
- * @position internal/vibe-tests/src/runner.ts
- */
-
-import Anthropic from '@anthropic-ai/sdk';
-import * as fs from 'node:fs';
-import * as path from 'node:path';
-import type {
- TestPrompt,
- TestResult,
- TestSet,
- Turn,
- RunConfig,
- PersonaType,
- Evaluation,
-} from './types.js';
-import {evaluate} from './evaluator.js';
-import {stratifiedSample} from './sampling.js';
-import {
- generateIterationId,
- hashContent,
- estimateTokens,
- sleep,
- ensureDir,
- appendJsonl,
- readPromptFile,
- getPromptsDir,
- getResultsDir,
- progressBar,
- timestamp,
-} from './utils.js';
-
-const RATE_LIMIT_DELAY = 500; // ms between requests
-
-export interface RunResult {
- iterationId: string;
- results: TestResult[];
- promptsRun: string[];
- promptsTotal: number;
-}
-
-/**
- * Load persona prompt
- */
-function loadPersona(persona: PersonaType): string {
- const personaPath = path.join(getPromptsDir(), 'personas', `${persona}.md`);
- return readPromptFile(personaPath);
-}
-
-/**
- * Wrap user prompt with persona framing
- */
-function applyPersona(prompt: string, personaPrompt: string): string {
- return `${personaPrompt}
-
----
-
-Now, as this persona, make the following request:
-
-${prompt}`;
-}
-
-/**
- * Run a single test prompt (one-shot)
- */
-async function runOneShot(
- client: Anthropic,
- testPrompt: TestPrompt,
- skillDoc: string,
- personaPrompt: string,
- config: RunConfig,
-): Promise<{response: string; conversation: Turn[]; tokenUsage: number}> {
- const framedPrompt = applyPersona(testPrompt.prompt, personaPrompt);
-
- const response = await client.messages.create({
- model: config.model,
- max_tokens: 4096,
- system: skillDoc,
- messages: [{role: 'user', content: framedPrompt}],
- });
-
- const content = response.content[0];
- if (content.type !== 'text') {
- throw new Error('Expected text response');
- }
-
- const turn: Turn = {
- type: 'probe',
- prompt: testPrompt.prompt,
- response: content.text,
- };
-
- const tokenUsage =
- estimateTokens(skillDoc) +
- estimateTokens(framedPrompt) +
- estimateTokens(content.text);
-
- return {
- response: content.text,
- conversation: [turn],
- tokenUsage,
- };
-}
-
-/**
- * Run degradation curve test
- */
-async function runDegradation(
- client: Anthropic,
- testPrompt: TestPrompt,
- skillDoc: string,
- personaPrompt: string,
- fillerPrompts: string[],
- distractorPrompts: string[],
- recoveryContext: string,
- config: RunConfig,
-): Promise<{
- turns: Turn[];
- evaluations: Map<number, Evaluation>;
- tokenUsage: number;
-}> {
- const messages: Anthropic.MessageParam[] = [];
- const turns: Turn[] = [];
- const evaluations = new Map<number, Evaluation>();
- let tokenUsage = estimateTokens(skillDoc);
-
- // Helper to add a turn
- const addTurn = async (
- type: Turn['type'],
- prompt: string,
- ): Promise<string> => {
- const framedPrompt =
- type === 'probe' ? applyPersona(prompt, personaPrompt) : prompt;
-
- messages.push({role: 'user', content: framedPrompt});
- tokenUsage += estimateTokens(framedPrompt);
-
- await sleep(RATE_LIMIT_DELAY);
- const response = await client.messages.create({
- model: config.model,
- max_tokens: 4096,
- system: skillDoc,
- messages,
- });
-
- const content = response.content[0];
- if (content.type !== 'text') {
- throw new Error('Expected text response');
- }
-
- messages.push({role: 'assistant', content: content.text});
- tokenUsage += estimateTokens(content.text);
-
- turns.push({type, prompt, response: content.text});
- return content.text;
- };
-
- // Turn 0: Initial probe
- const initialResponse = await addTurn('probe', testPrompt.prompt);
- const eval0 = await evaluate(
- client,
- {
- userPrompt: testPrompt.prompt,
- llmResponse: initialResponse,
- expectedComponents: testPrompt.expectedComponents,
- skillDoc,
- },
- config.model,
- );
- turns[0].evaluation = eval0;
- evaluations.set(0, eval0);
-
- // Turns 1-5: Filler
- for (let i = 0; i < 5; i++) {
- const fillerPrompt =
- fillerPrompts[Math.floor(Math.random() * fillerPrompts.length)];
- await addTurn('filler', fillerPrompt);
- }
-
- // Turn 6: Re-probe with natural framing
- const reprobe1 = `Actually, let me revisit this. ${testPrompt.prompt}`;
- const response6 = await addTurn('probe', reprobe1);
- const eval6 = await evaluate(
- client,
- {
- userPrompt: testPrompt.prompt,
- llmResponse: response6,
- expectedComponents: testPrompt.expectedComponents,
- skillDoc,
- },
- config.model,
- );
- turns[turns.length - 1].evaluation = eval6;
- evaluations.set(6, eval6);
-
- // Turn 7: Distractor
- const distractor =
- distractorPrompts[Math.floor(Math.random() * distractorPrompts.length)];
- await addTurn('distractor', distractor);
-
- // Turn 8: Re-probe after distractor
- const reprobe2 = `I want to redo that earlier request. ${testPrompt.prompt}`;
- const response8 = await addTurn('probe', reprobe2);
- const eval8 = await evaluate(
- client,
- {
- userPrompt: testPrompt.prompt,
- llmResponse: response8,
- expectedComponents: testPrompt.expectedComponents,
- skillDoc,
- },
- config.model,
- );
- turns[turns.length - 1].evaluation = eval8;
- evaluations.set(8, eval8);
-
- // Turn 9: Recovery (re-inject partial skill doc)
- const recovery = `${recoveryContext}\n\n${skillDoc.slice(0, 2000)}...`;
- await addTurn('recovery', recovery);
-
- // Turn 10: Final re-probe after recovery
- const reprobe3 = `Let me try that again. ${testPrompt.prompt}`;
- const response10 = await addTurn('probe', reprobe3);
- const eval10 = await evaluate(
- client,
- {
- userPrompt: testPrompt.prompt,
- llmResponse: response10,
- expectedComponents: testPrompt.expectedComponents,
- skillDoc,
- },
- config.model,
- );
- turns[turns.length - 1].evaluation = eval10;
- evaluations.set(10, eval10);
-
- return {turns, evaluations, tokenUsage};
-}
-
-/**
- * Run tests with the given configuration
- */
-export async function runTests(
- testSet: TestSet,
- config: RunConfig,
-): Promise<RunResult> {
- const client = new Anthropic();
- const iterationId = generateIterationId();
- const skillDoc = fs.readFileSync(config.skillDocPath, 'utf-8');
- const personaPrompt = loadPersona(config.persona);
-
- // Select prompts
- let prompts: TestPrompt[];
- if (config.holdout && testSet.holdout) {
- prompts = testSet.holdout;
- } else if (config.sample && config.sample < testSet.prompts.length) {
- prompts = stratifiedSample(testSet.prompts, config.sample);
- } else {
- prompts = testSet.prompts;
- }
-
- // Setup results directory
- const resultsDir = path.join(getResultsDir(), iterationId);
- ensureDir(resultsDir);
- const runsPath = path.join(resultsDir, 'runs.jsonl');
-
- const results: TestResult[] = [];
- const promptsRun: string[] = [];
-
- console.log(`\nRunning vibeability tests...`);
- console.log(` Protocol: ${config.protocol}`);
- console.log(` Persona: ${config.persona}`);
- console.log(` Model: ${config.model}`);
- console.log(
- ` Test set: ${testSet.name} (${prompts.length}${config.sample ? ` sampled from ${testSet.prompts.length}` : ''} prompts)`,
- );
- console.log(` Iteration: ${iterationId}\n`);
-
- for (let i = 0; i < prompts.length; i++) {
- const testPrompt = prompts[i];
- promptsRun.push(testPrompt.id);
-
- process.stdout.write(` ${progressBar(i + 1, prompts.length)}\r`);
-
- try {
- if (config.protocol === 'one-shot') {
- const {response, conversation, tokenUsage} = await runOneShot(
- client,
- testPrompt,
- skillDoc,
- personaPrompt,
- config,
- );
-
- const evaluation = await evaluate(
- client,
- {
- userPrompt: testPrompt.prompt,
- llmResponse: response,
- expectedComponents: testPrompt.expectedComponents,
- skillDoc,
- },
- config.model,
- );
-
- conversation[0].evaluation = evaluation;
-
- const result: TestResult = {
- id: `${iterationId}-${testPrompt.id}`,
- timestamp: timestamp(),
- systemVersion: hashContent(skillDoc),
- model: config.model,
- persona: config.persona,
- promptCategory: testPrompt.category,
- trajectoryDepth: 0,
- prompt: testPrompt.prompt,
- response,
- evaluation,
- fullConversation: conversation,
- contextWindowUsage: tokenUsage,
- };
-
- results.push(result);
- appendJsonl(runsPath, result);
- } else if (config.protocol === 'degradation') {
- const {turns, evaluations, tokenUsage} = await runDegradation(
- client,
- testPrompt,
- skillDoc,
- personaPrompt,
- testSet.fillerPrompts ?? [],
- testSet.distractorPrompts ?? [],
- testSet.recoveryContext ?? '',
- config,
- );
-
- // Create results for each evaluation point
- for (const [depth, evaluation] of evaluations) {
- const result: TestResult = {
- id: `${iterationId}-${testPrompt.id}-t${depth}`,
- timestamp: timestamp(),
- systemVersion: hashContent(skillDoc),
- model: config.model,
- persona: config.persona,
- promptCategory: testPrompt.category,
- trajectoryDepth: depth,
- prompt: testPrompt.prompt,
- response: turns[depth]?.response ?? '',
- evaluation,
- fullConversation: turns,
- contextWindowUsage: tokenUsage,
- };
-
- results.push(result);
- appendJsonl(runsPath, result);
- }
- }
- } catch (error) {
- console.error(`\nError on prompt ${testPrompt.id}:`, error);
- }
-
- await sleep(RATE_LIMIT_DELAY);
- }
-
- console.log(`\n`);
-
- return {
- iterationId,
- results,
- promptsRun,
- promptsTotal: testSet.prompts.length,
- };
-}
diff --git a/internal/vibe-tests/src/screenshot-previews.ts b/internal/vibe-tests/src/screenshot-previews.ts
new file mode 100644
index 0000000000..3074110612
--- /dev/null
+++ b/internal/vibe-tests/src/screenshot-previews.ts
@@ -0,0 +1,283 @@
+#!/usr/bin/env node
+/**
+ * @file Screenshot pre-built preview HTML files using Playwright
+ * @input Pre-built preview HTML files from build-previews.ts
+ * @output PNG screenshots at multiple viewport/theme combos
+ * @position internal/vibe-tests/src/screenshot-previews.ts
+ *
+ * Unlike screenshot.ts (which runs a Vite dev server), this script
+ * works with pre-built self-contained HTML files. Designed to run in
+ * GitHub Actions where Playwright is installed but XDS source isn't needed
+ * for rendering.
+ *
+ * Usage:
+ * tsx src/screenshot-previews.ts --iterations 8734233a,d4ff8c2c
+ * tsx src/screenshot-previews.ts --iterations 8734233a --prompts tc-4,tc-7
+ */
+
+import * as fs from 'node:fs';
+import * as path from 'node:path';
+import * as http from 'node:http';
+import {getResultsDir, ensureDir, writeJson} from './utils.js';
+
+const VIEWPORTS = {
+ desktop: {width: 1280, height: 800},
+ mobile: {width: 375, height: 812},
+};
+
+const THEMES = ['light', 'dark'] as const;
+
+function parseArgs(): {iterations: string[]; prompts?: string[]} {
+ const args = process.argv.slice(2);
+ let iterations: string[] = [];
+ let prompts: string[] | undefined;
+
+ for (let i = 0; i < args.length; i++) {
+ if (args[i] === '--iterations' && args[i + 1]) {
+ iterations = args[++i].split(',');
+ } else if (args[i] === '--prompts' && args[i + 1]) {
+ prompts = args[++i].split(',');
+ }
+ }
+
+ if (iterations.length === 0) {
+ console.error(
+ 'Usage: tsx src/screenshot-previews.ts --iterations <id1,id2,...> [--prompts <p1,p2,...>]',
+ );
+ process.exit(1);
+ }
+
+ return {iterations, prompts};
+}
+
+/**
+ * Serve a directory of static files on a random port.
+ * Returns the base URL and a close function.
+ */
+function serveStatic(
+ dir: string,
+): Promise<{url: string; close: () => Promise<void>}> {
+ return new Promise((resolve, reject) => {
+ const server = http.createServer((req, res) => {
+ const urlPath = decodeURIComponent(req.url || '/');
+ const filePath = path.join(dir, urlPath);
+
+ if (!fs.existsSync(filePath)) {
+ res.writeHead(404);
+ res.end('Not found');
+ return;
+ }
+
+ const ext = path.extname(filePath).toLowerCase();
+ const contentTypes: Record<string, string> = {
+ '.html': 'text/html',
+ '.css': 'text/css',
+ '.js': 'application/javascript',
+ '.png': 'image/png',
+ '.svg': 'image/svg+xml',
+ '.json': 'application/json',
+ };
+
+ const content = fs.readFileSync(filePath);
+ res.writeHead(200, {'Content-Type': contentTypes[ext] || 'text/plain'});
+ res.end(content);
+ });
+
+ server.listen(0, '127.0.0.1', () => {
+ const addr = server.address();
+ if (!addr || typeof addr === 'string') {
+ reject(new Error('Failed to get server address'));
+ return;
+ }
+ resolve({
+ url: `http://127.0.0.1:${addr.port}`,
+ close: () =>
+ new Promise<void>(r => {
+ server.close(() => r());
+ }),
+ });
+ });
+
+ server.on('error', reject);
+ });
+}
+
+/**
+ * Apply a color-scheme to the page for dark mode screenshots.
+ * For self-contained HTML previews, we inject prefers-color-scheme
+ * emulation via Playwright's CDP.
+ */
+async function setTheme(
+ page: import('playwright').Page,
+ theme: 'light' | 'dark',
+) {
+ await page.emulateMedia({colorScheme: theme});
+}
+
+async function main() {
+ const {iterations, prompts} = parseArgs();
+ const resultsDir = getResultsDir();
+
+ // Import playwright
+ let chromium: typeof import('playwright').chromium;
+ try {
+ const pw = await import('playwright');
+ chromium = pw.chromium;
+ } catch {
+ console.error(
+ 'Playwright is not installed. Run:\n' +
+ ' npx playwright install chromium --with-deps\n',
+ );
+ process.exit(1);
+ }
+
+ const browser = await chromium.launch();
+ let totalScreenshots = 0;
+
+ try {
+ for (const iterationId of iterations) {
+ const iterDir = path.join(resultsDir, iterationId);
+ const previewsDir = path.join(iterDir, 'previews');
+ const screenshotsDir = path.join(iterDir, 'screenshots');
+
+ if (!fs.existsSync(previewsDir)) {
+ console.error(
+ ` ⚠ No previews directory for ${iterationId}, skipping`,
+ );
+ continue;
+ }
+
+ // Read the preview manifest to find all HTML files
+ const manifestPath = path.join(previewsDir, 'manifest.json');
+ const previewFiles: Array<{promptId: string; target: string; path: string}> =
+ [];
+
+ if (fs.existsSync(manifestPath)) {
+ const manifest = JSON.parse(fs.readFileSync(manifestPath, 'utf-8'));
+ for (const [promptId, targets] of Object.entries(manifest)) {
+ if (prompts && !prompts.includes(promptId)) continue;
+ for (const [target, relPath] of Object.entries(
+ targets as Record<string, string>,
+ )) {
+ const fullPath = path.join(iterDir, relPath);
+ if (fs.existsSync(fullPath)) {
+ previewFiles.push({promptId, target, path: fullPath});
+ }
+ }
+ }
+ } else {
+ // No manifest — scan for HTML files directly
+ const scanDir = (dir: string) => {
+ if (!fs.existsSync(dir)) return;
+ for (const entry of fs.readdirSync(dir, {withFileTypes: true})) {
+ if (entry.isDirectory()) {
+ scanDir(path.join(dir, entry.name));
+ } else if (entry.name.endsWith('.html')) {
+ const promptId = path.basename(path.dirname(path.join(dir, entry.name)));
+ const target = path.basename(entry.name, '.html');
+ previewFiles.push({
+ promptId,
+ target,
+ path: path.join(dir, entry.name),
+ });
+ }
+ }
+ };
+ scanDir(previewsDir);
+ }
+
+ if (previewFiles.length === 0) {
+ console.error(` ⚠ No preview HTML files found for ${iterationId}`);
+ continue;
+ }
+
+ // Serve the previews directory
+ const server = await serveStatic(path.dirname(previewsDir));
+ ensureDir(screenshotsDir);
+
+ console.log(
+ `\n📸 Screenshotting ${previewFiles.length} preview(s) for ${iterationId}...\n`,
+ );
+
+ const screenshotManifest: Record<
+ string,
+ Record<string, Record<string, Record<string, string>>>
+ > = {};
+
+ for (const preview of previewFiles) {
+ // Compute the URL relative to the served directory
+ const relPath = path.relative(
+ path.dirname(previewsDir),
+ preview.path,
+ );
+
+ for (const [viewportName, viewport] of Object.entries(VIEWPORTS)) {
+ for (const theme of THEMES) {
+ const page = await browser.newPage({viewport});
+ await setTheme(page, theme);
+
+ const url = `${server.url}/${relPath}`;
+ try {
+ await page.goto(url, {
+ waitUntil: 'networkidle',
+ timeout: 30000,
+ });
+ } catch {
+ // Fallback: wait for load instead of networkidle
+ // (some pages have long-lived connections)
+ await page.goto(url, {waitUntil: 'load', timeout: 15000});
+ }
+
+ // Wait for fonts and rendering
+ await page.waitForTimeout(1000);
+
+ const filename = `${preview.promptId}-${preview.target}-${viewportName}-${theme}.png`;
+ const screenshotPath = path.join(screenshotsDir, filename);
+ await page.screenshot({path: screenshotPath, fullPage: true});
+ await page.close();
+
+ // Track in manifest
+ if (!screenshotManifest[preview.promptId]) {
+ screenshotManifest[preview.promptId] = {};
+ }
+ if (!screenshotManifest[preview.promptId][preview.target]) {
+ screenshotManifest[preview.promptId][preview.target] = {};
+ }
+ if (
+ !screenshotManifest[preview.promptId][preview.target][
+ viewportName
+ ]
+ ) {
+ screenshotManifest[preview.promptId][preview.target][
+ viewportName
+ ] = {};
+ }
+ screenshotManifest[preview.promptId][preview.target][viewportName][
+ theme
+ ] = filename;
+
+ totalScreenshots++;
+ console.log(` ✓ ${filename}`);
+ }
+ }
+ }
+
+ // Write screenshot manifest
+ writeJson(
+ path.join(screenshotsDir, 'manifest.json'),
+ screenshotManifest,
+ );
+
+ await server.close();
+ }
+ } finally {
+ await browser.close();
+ }
+
+ console.log(`\n✅ Captured ${totalScreenshots} screenshot(s)`);
+}
+
+main().catch(err => {
+ console.error(err);
+ process.exit(1);
+});