Skip to content

An agentic skills framework & software development methodology that works.

Add comprehensive testing documentation

Jesse Vincent9464a51779a0
diff --git a/docs/testing.md b/docs/testing.md
new file mode 100644
index 000000000..6f87afed7
--- /dev/null
+++ b/docs/testing.md
@@ -0,0 +1,303 @@
+# Testing Superpowers Skills
+
+This document describes how to test Superpowers skills, particularly the integration tests for complex skills like `subagent-driven-development`.
+
+## Overview
+
+Testing skills that involve subagents, workflows, and complex interactions requires running actual Claude Code sessions in headless mode and verifying their behavior through session transcripts.
+
+## Test Structure
+
+```
+tests/
+├── claude-code/
+│ ├── test-helpers.sh # Shared test utilities
+│ ├── test-subagent-driven-development-integration.sh
+│ ├── analyze-token-usage.py # Token analysis tool
+│ └── run-skill-tests.sh # Test runner (if exists)
+```
+
+## Running Tests
+
+### Integration Tests
+
+Integration tests execute real Claude Code sessions with actual skills:
+
+```bash
+# Run the subagent-driven-development integration test
+cd tests/claude-code
+./test-subagent-driven-development-integration.sh
+```
+
+**Note:** Integration tests can take 10-30 minutes as they execute real implementation plans with multiple subagents.
+
+### Requirements
+
+- Must run from the **superpowers plugin directory** (not from temp directories)
+- Claude Code must be installed and available as `claude` command
+- Local dev marketplace must be enabled: `"superpowers@superpowers-dev": true` in `~/.claude/settings.json`
+
+## Integration Test: subagent-driven-development
+
+### What It Tests
+
+The integration test verifies the `subagent-driven-development` skill correctly:
+
+1. **Plan Loading**: Reads the plan once at the beginning
+2. **Full Task Text**: Provides complete task descriptions to subagents (doesn't make them read files)
+3. **Self-Review**: Ensures subagents perform self-review before reporting
+4. **Review Order**: Runs spec compliance review before code quality review
+5. **Review Loops**: Uses review loops when issues are found
+6. **Independent Verification**: Spec reviewer reads code independently, doesn't trust implementer reports
+
+### How It Works
+
+1. **Setup**: Creates a temporary Node.js project with a minimal implementation plan
+2. **Execution**: Runs Claude Code in headless mode with the skill
+3. **Verification**: Parses the session transcript (`.jsonl` file) to verify:
+ - Skill tool was invoked
+ - Subagents were dispatched (Task tool)
+ - TodoWrite was used for tracking
+ - Implementation files were created
+ - Tests pass
+ - Git commits show proper workflow
+4. **Token Analysis**: Shows token usage breakdown by subagent
+
+### Test Output
+
+```
+========================================
+ Integration Test: subagent-driven-development
+========================================
+
+Test project: /tmp/tmp.xyz123
+
+=== Verification Tests ===
+
+Test 1: Skill tool invoked...
+ [PASS] subagent-driven-development skill was invoked
+
+Test 2: Subagents dispatched...
+ [PASS] 7 subagents dispatched
+
+Test 3: Task tracking...
+ [PASS] TodoWrite used 5 time(s)
+
+Test 6: Implementation verification...
+ [PASS] src/math.js created
+ [PASS] add function exists
+ [PASS] multiply function exists
+ [PASS] test/math.test.js created
+ [PASS] Tests pass
+
+Test 7: Git commit history...
+ [PASS] Multiple commits created (3 total)
+
+Test 8: No extra features added...
+ [PASS] No extra features added
+
+=========================================
+ Token Usage Analysis
+=========================================
+
+Usage Breakdown:
+----------------------------------------------------------------------------------------------------
+Agent Description Msgs Input Output Cache Cost
+----------------------------------------------------------------------------------------------------
+main Main session (coordinator) 34 27 3,996 1,213,703 $ 4.09
+3380c209 implementing Task 1: Create Add Function 1 2 787 24,989 $ 0.09
+34b00fde implementing Task 2: Create Multiply Function 1 4 644 25,114 $ 0.09
+3801a732 reviewing whether an implementation matches... 1 5 703 25,742 $ 0.09
+4c142934 doing a final code review... 1 6 854 25,319 $ 0.09
+5f017a42 a code reviewer. Review Task 2... 1 6 504 22,949 $ 0.08
+a6b7fbe4 a code reviewer. Review Task 1... 1 6 515 22,534 $ 0.08
+f15837c0 reviewing whether an implementation matches... 1 6 416 22,485 $ 0.07
+----------------------------------------------------------------------------------------------------
+
+TOTALS:
+ Total messages: 41
+ Input tokens: 62
+ Output tokens: 8,419
+ Cache creation tokens: 132,742
+ Cache read tokens: 1,382,835
+
+ Total input (incl cache): 1,515,639
+ Total tokens: 1,524,058
+
+ Estimated cost: $4.67
+ (at $3/$15 per M tokens for input/output)
+
+========================================
+ Test Summary
+========================================
+
+STATUS: PASSED
+```
+
+## Token Analysis Tool
+
+### Usage
+
+Analyze token usage from any Claude Code session:
+
+```bash
+python3 tests/claude-code/analyze-token-usage.py ~/.claude/projects/<project-dir>/<session-id>.jsonl
+```
+
+### Finding Session Files
+
+Session transcripts are stored in `~/.claude/projects/` with the working directory path encoded:
+
+```bash
+# Example for /Users/jesse/Documents/GitHub/superpowers/superpowers
+SESSION_DIR="$HOME/.claude/projects/-Users-jesse-Documents-GitHub-superpowers-superpowers"
+
+# Find recent sessions
+ls -lt "$SESSION_DIR"/*.jsonl | head -5
+```
+
+### What It Shows
+
+- **Main session usage**: Token usage by the coordinator (you or main Claude instance)
+- **Per-subagent breakdown**: Each Task invocation with:
+ - Agent ID
+ - Description (extracted from prompt)
+ - Message count
+ - Input/output tokens
+ - Cache usage
+ - Estimated cost
+- **Totals**: Overall token usage and cost estimate
+
+### Understanding the Output
+
+- **High cache reads**: Good - means prompt caching is working
+- **High input tokens on main**: Expected - coordinator has full context
+- **Similar costs per subagent**: Expected - each gets similar task complexity
+- **Cost per task**: Typical range is $0.05-$0.15 per subagent depending on task
+
+## Troubleshooting
+
+### Skills Not Loading
+
+**Problem**: Skill not found when running headless tests
+
+**Solutions**:
+1. Ensure you're running FROM the superpowers directory: `cd /path/to/superpowers && tests/...`
+2. Check `~/.claude/settings.json` has `"superpowers@superpowers-dev": true` in `enabledPlugins`
+3. Verify skill exists in `skills/` directory
+
+### Permission Errors
+
+**Problem**: Claude blocked from writing files or accessing directories
+
+**Solutions**:
+1. Use `--permission-mode bypassPermissions` flag
+2. Use `--add-dir /path/to/temp/dir` to grant access to test directories
+3. Check file permissions on test directories
+
+### Test Timeouts
+
+**Problem**: Test takes too long and times out
+
+**Solutions**:
+1. Increase timeout: `timeout 1800 claude ...` (30 minutes)
+2. Check for infinite loops in skill logic
+3. Review subagent task complexity
+
+### Session File Not Found
+
+**Problem**: Can't find session transcript after test run
+
+**Solutions**:
+1. Check the correct project directory in `~/.claude/projects/`
+2. Use `find ~/.claude/projects -name "*.jsonl" -mmin -60` to find recent sessions
+3. Verify test actually ran (check for errors in test output)
+
+## Writing New Integration Tests
+
+### Template
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/test-helpers.sh"
+
+# Create test project
+TEST_PROJECT=$(create_test_project)
+trap "cleanup_test_project $TEST_PROJECT" EXIT
+
+# Set up test files...
+cd "$TEST_PROJECT"
+
+# Run Claude with skill
+PROMPT="Your test prompt here"
+cd "$SCRIPT_DIR/../.." && timeout 1800 claude -p "$PROMPT" \
+ --allowed-tools=all \
+ --add-dir "$TEST_PROJECT" \
+ --permission-mode bypassPermissions \
+ 2>&1 | tee output.txt
+
+# Find and analyze session
+WORKING_DIR_ESCAPED=$(echo "$SCRIPT_DIR/../.." | sed 's/\\//-/g' | sed 's/^-//')
+SESSION_DIR="$HOME/.claude/projects/$WORKING_DIR_ESCAPED"
+SESSION_FILE=$(find "$SESSION_DIR" -name "*.jsonl" -type f -mmin -60 | sort -r | head -1)
+
+# Verify behavior by parsing session transcript
+if grep -q '"name":"Skill".*"skill":"your-skill-name"' "$SESSION_FILE"; then
+ echo "[PASS] Skill was invoked"
+fi
+
+# Show token analysis
+python3 "$SCRIPT_DIR/analyze-token-usage.py" "$SESSION_FILE"
+```
+
+### Best Practices
+
+1. **Always cleanup**: Use trap to cleanup temp directories
+2. **Parse transcripts**: Don't grep user-facing output - parse the `.jsonl` session file
+3. **Grant permissions**: Use `--permission-mode bypassPermissions` and `--add-dir`
+4. **Run from plugin dir**: Skills only load when running from the superpowers directory
+5. **Show token usage**: Always include token analysis for cost visibility
+6. **Test real behavior**: Verify actual files created, tests passing, commits made
+
+## Session Transcript Format
+
+Session transcripts are JSONL (JSON Lines) files where each line is a JSON object representing a message or tool result.
+
+### Key Fields
+
+```json
+{
+ "type": "assistant",
+ "message": {
+ "content": [...],
+ "usage": {
+ "input_tokens": 27,
+ "output_tokens": 3996,
+ "cache_read_input_tokens": 1213703
+ }
+ }
+}
+```
+
+### Tool Results
+
+```json
+{
+ "type": "user",
+ "toolUseResult": {
+ "agentId": "3380c209",
+ "usage": {
+ "input_tokens": 2,
+ "output_tokens": 787,
+ "cache_read_input_tokens": 24989
+ },
+ "prompt": "You are implementing Task 1...",
+ "content": [{"type": "text", "text": "..."}]
+ }
+}
+```
+
+The `agentId` field links to subagent sessions, and the `usage` field contains token usage for that specific subagent invocation.