An agentic skills framework & software development methodology that works.
Add comprehensive testing documentation
Jesse Vincent9464a51779a0
diff --git a/docs/testing.md b/docs/testing.mdnew file mode 100644index 000000000..6f87afed7--- /dev/null+++ b/docs/testing.md@@ -0,0 +1,303 @@+# Testing Superpowers Skills++This document describes how to test Superpowers skills, particularly the integration tests for complex skills like `subagent-driven-development`.++## Overview++Testing skills that involve subagents, workflows, and complex interactions requires running actual Claude Code sessions in headless mode and verifying their behavior through session transcripts.++## Test Structure++```+tests/+├── claude-code/+│ ├── test-helpers.sh # Shared test utilities+│ ├── test-subagent-driven-development-integration.sh+│ ├── analyze-token-usage.py # Token analysis tool+│ └── run-skill-tests.sh # Test runner (if exists)+```++## Running Tests++### Integration Tests++Integration tests execute real Claude Code sessions with actual skills:++```bash+# Run the subagent-driven-development integration test+cd tests/claude-code+./test-subagent-driven-development-integration.sh+```++**Note:** Integration tests can take 10-30 minutes as they execute real implementation plans with multiple subagents.++### Requirements++- Must run from the **superpowers plugin directory** (not from temp directories)+- Claude Code must be installed and available as `claude` command+- Local dev marketplace must be enabled: `"superpowers@superpowers-dev": true` in `~/.claude/settings.json`++## Integration Test: subagent-driven-development++### What It Tests++The integration test verifies the `subagent-driven-development` skill correctly:++1. **Plan Loading**: Reads the plan once at the beginning+2. **Full Task Text**: Provides complete task descriptions to subagents (doesn't make them read files)+3. **Self-Review**: Ensures subagents perform self-review before reporting+4. **Review Order**: Runs spec compliance review before code quality review+5. **Review Loops**: Uses review loops when issues are found+6. **Independent Verification**: Spec reviewer reads code independently, doesn't trust implementer reports++### How It Works++1. **Setup**: Creates a temporary Node.js project with a minimal implementation plan+2. **Execution**: Runs Claude Code in headless mode with the skill+3. **Verification**: Parses the session transcript (`.jsonl` file) to verify:+ - Skill tool was invoked+ - Subagents were dispatched (Task tool)+ - TodoWrite was used for tracking+ - Implementation files were created+ - Tests pass+ - Git commits show proper workflow+4. **Token Analysis**: Shows token usage breakdown by subagent++### Test Output++```+========================================+ Integration Test: subagent-driven-development+========================================++Test project: /tmp/tmp.xyz123++=== Verification Tests ===++Test 1: Skill tool invoked...+ [PASS] subagent-driven-development skill was invoked++Test 2: Subagents dispatched...+ [PASS] 7 subagents dispatched++Test 3: Task tracking...+ [PASS] TodoWrite used 5 time(s)++Test 6: Implementation verification...+ [PASS] src/math.js created+ [PASS] add function exists+ [PASS] multiply function exists+ [PASS] test/math.test.js created+ [PASS] Tests pass++Test 7: Git commit history...+ [PASS] Multiple commits created (3 total)++Test 8: No extra features added...+ [PASS] No extra features added++=========================================+ Token Usage Analysis+=========================================++Usage Breakdown:+----------------------------------------------------------------------------------------------------+Agent Description Msgs Input Output Cache Cost+----------------------------------------------------------------------------------------------------+main Main session (coordinator) 34 27 3,996 1,213,703 $ 4.09+3380c209 implementing Task 1: Create Add Function 1 2 787 24,989 $ 0.09+34b00fde implementing Task 2: Create Multiply Function 1 4 644 25,114 $ 0.09+3801a732 reviewing whether an implementation matches... 1 5 703 25,742 $ 0.09+4c142934 doing a final code review... 1 6 854 25,319 $ 0.09+5f017a42 a code reviewer. Review Task 2... 1 6 504 22,949 $ 0.08+a6b7fbe4 a code reviewer. Review Task 1... 1 6 515 22,534 $ 0.08+f15837c0 reviewing whether an implementation matches... 1 6 416 22,485 $ 0.07+----------------------------------------------------------------------------------------------------++TOTALS:+ Total messages: 41+ Input tokens: 62+ Output tokens: 8,419+ Cache creation tokens: 132,742+ Cache read tokens: 1,382,835++ Total input (incl cache): 1,515,639+ Total tokens: 1,524,058++ Estimated cost: $4.67+ (at $3/$15 per M tokens for input/output)++========================================+ Test Summary+========================================++STATUS: PASSED+```++## Token Analysis Tool++### Usage++Analyze token usage from any Claude Code session:++```bash+python3 tests/claude-code/analyze-token-usage.py ~/.claude/projects/<project-dir>/<session-id>.jsonl+```++### Finding Session Files++Session transcripts are stored in `~/.claude/projects/` with the working directory path encoded:++```bash+# Example for /Users/jesse/Documents/GitHub/superpowers/superpowers+SESSION_DIR="$HOME/.claude/projects/-Users-jesse-Documents-GitHub-superpowers-superpowers"++# Find recent sessions+ls -lt "$SESSION_DIR"/*.jsonl | head -5+```++### What It Shows++- **Main session usage**: Token usage by the coordinator (you or main Claude instance)+- **Per-subagent breakdown**: Each Task invocation with:+ - Agent ID+ - Description (extracted from prompt)+ - Message count+ - Input/output tokens+ - Cache usage+ - Estimated cost+- **Totals**: Overall token usage and cost estimate++### Understanding the Output++- **High cache reads**: Good - means prompt caching is working+- **High input tokens on main**: Expected - coordinator has full context+- **Similar costs per subagent**: Expected - each gets similar task complexity+- **Cost per task**: Typical range is $0.05-$0.15 per subagent depending on task++## Troubleshooting++### Skills Not Loading++**Problem**: Skill not found when running headless tests++**Solutions**:+1. Ensure you're running FROM the superpowers directory: `cd /path/to/superpowers && tests/...`+2. Check `~/.claude/settings.json` has `"superpowers@superpowers-dev": true` in `enabledPlugins`+3. Verify skill exists in `skills/` directory++### Permission Errors++**Problem**: Claude blocked from writing files or accessing directories++**Solutions**:+1. Use `--permission-mode bypassPermissions` flag+2. Use `--add-dir /path/to/temp/dir` to grant access to test directories+3. Check file permissions on test directories++### Test Timeouts++**Problem**: Test takes too long and times out++**Solutions**:+1. Increase timeout: `timeout 1800 claude ...` (30 minutes)+2. Check for infinite loops in skill logic+3. Review subagent task complexity++### Session File Not Found++**Problem**: Can't find session transcript after test run++**Solutions**:+1. Check the correct project directory in `~/.claude/projects/`+2. Use `find ~/.claude/projects -name "*.jsonl" -mmin -60` to find recent sessions+3. Verify test actually ran (check for errors in test output)++## Writing New Integration Tests++### Template++```bash+#!/usr/bin/env bash+set -euo pipefail++SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"+source "$SCRIPT_DIR/test-helpers.sh"++# Create test project+TEST_PROJECT=$(create_test_project)+trap "cleanup_test_project $TEST_PROJECT" EXIT++# Set up test files...+cd "$TEST_PROJECT"++# Run Claude with skill+PROMPT="Your test prompt here"+cd "$SCRIPT_DIR/../.." && timeout 1800 claude -p "$PROMPT" \+ --allowed-tools=all \+ --add-dir "$TEST_PROJECT" \+ --permission-mode bypassPermissions \+ 2>&1 | tee output.txt++# Find and analyze session+WORKING_DIR_ESCAPED=$(echo "$SCRIPT_DIR/../.." | sed 's/\\//-/g' | sed 's/^-//')+SESSION_DIR="$HOME/.claude/projects/$WORKING_DIR_ESCAPED"+SESSION_FILE=$(find "$SESSION_DIR" -name "*.jsonl" -type f -mmin -60 | sort -r | head -1)++# Verify behavior by parsing session transcript+if grep -q '"name":"Skill".*"skill":"your-skill-name"' "$SESSION_FILE"; then+ echo "[PASS] Skill was invoked"+fi++# Show token analysis+python3 "$SCRIPT_DIR/analyze-token-usage.py" "$SESSION_FILE"+```++### Best Practices++1. **Always cleanup**: Use trap to cleanup temp directories+2. **Parse transcripts**: Don't grep user-facing output - parse the `.jsonl` session file+3. **Grant permissions**: Use `--permission-mode bypassPermissions` and `--add-dir`+4. **Run from plugin dir**: Skills only load when running from the superpowers directory+5. **Show token usage**: Always include token analysis for cost visibility+6. **Test real behavior**: Verify actual files created, tests passing, commits made++## Session Transcript Format++Session transcripts are JSONL (JSON Lines) files where each line is a JSON object representing a message or tool result.++### Key Fields++```json+{+ "type": "assistant",+ "message": {+ "content": [...],+ "usage": {+ "input_tokens": 27,+ "output_tokens": 3996,+ "cache_read_input_tokens": 1213703+ }+ }+}+```++### Tool Results++```json+{+ "type": "user",+ "toolUseResult": {+ "agentId": "3380c209",+ "usage": {+ "input_tokens": 2,+ "output_tokens": 787,+ "cache_read_input_tokens": 24989+ },+ "prompt": "You are implementing Task 1...",+ "content": [{"type": "text", "text": "..."}]+ }+}+```++The `agentId` field links to subagent sessions, and the `usage` field contains token usage for that specific subagent invocation.