An agentic skills framework & software development methodology that works.
Add Claude Code skills test framework
Jesse Vincent51a171cd141f
diff --git a/tests/claude-code/README.md b/tests/claude-code/README.mdnew file mode 100644index 000000000..177f791a1--- /dev/null+++ b/tests/claude-code/README.md@@ -0,0 +1,128 @@+# Claude Code Skills Tests++Automated tests for superpowers skills using Claude Code CLI.++## Overview++This test suite verifies that skills are loaded correctly and Claude follows them as expected. Tests invoke Claude Code in headless mode (`claude -p`) and verify the behavior.++## Requirements++- Claude Code CLI installed and in PATH (`claude --version` should work)+- Local superpowers plugin installed (see main README for installation)++## Running Tests++### Run all tests:+```bash+./run-skill-tests.sh+```++### Run specific test:+```bash+./run-skill-tests.sh --test test-subagent-driven-development.sh+```++### Run with verbose output:+```bash+./run-skill-tests.sh --verbose+```++### Set custom timeout (default 300s):+```bash+./run-skill-tests.sh --timeout 600+```++## Test Structure++### test-helpers.sh+Common functions for skills testing:+- `run_claude "prompt" [timeout]` - Run Claude with prompt+- `assert_contains output pattern name` - Verify pattern exists+- `assert_not_contains output pattern name` - Verify pattern absent+- `assert_count output pattern count name` - Verify exact count+- `assert_order output pattern_a pattern_b name` - Verify order+- `create_test_project` - Create temp test directory+- `create_test_plan project_dir` - Create sample plan file++### Test Files++Each test file:+1. Sources `test-helpers.sh`+2. Runs Claude Code with specific prompts+3. Verifies expected behavior using assertions+4. Returns 0 on success, non-zero on failure++## Example Test++```bash+#!/usr/bin/env bash+set -euo pipefail++SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"+source "$SCRIPT_DIR/test-helpers.sh"++echo "=== Test: My Skill ==="++# Ask Claude about the skill+output=$(run_claude "What does the my-skill skill do?" 30)++# Verify response+assert_contains "$output" "expected behavior" "Skill describes behavior"++echo "=== All tests passed ==="+```++## Current Tests++### test-subagent-driven-development.sh+Tests the updated subagent-driven-development workflow:+- Skill loading+- Workflow ordering (spec compliance before code quality)+- Self-review requirements+- Plan reading efficiency (read once)+- Spec compliance reviewer skepticism+- Review loops+- Task context provision (full text, not file reading)++## Adding New Tests++1. Create new test file: `test-<skill-name>.sh`+2. Source test-helpers.sh+3. Write tests using `run_claude` and assertions+4. Add to test list in `run-skill-tests.sh`+5. Make executable: `chmod +x test-<skill-name>.sh`++## Timeout Considerations++- Default timeout: 5 minutes per test+- Claude Code may take time to respond+- Adjust with `--timeout` if needed+- Tests should be focused to avoid long runs++## Debugging Failed Tests++With `--verbose`, you'll see full Claude output:+```bash+./run-skill-tests.sh --verbose --test test-subagent-driven-development.sh+```++Without verbose, only failures show output.++## CI/CD Integration++To run in CI:+```bash+# Run with explicit timeout for CI environments+./run-skill-tests.sh --timeout 900++# Exit code 0 = success, non-zero = failure+```++## Notes++- Tests verify skill *instructions*, not full execution+- Full workflow tests would be very slow+- Focus on verifying key skill requirements+- Tests should be deterministic+- Avoid testing implementation detailsdiff --git a/tests/claude-code/run-skill-tests.sh b/tests/claude-code/run-skill-tests.shnew file mode 100755index 000000000..a50d71d69--- /dev/null+++ b/tests/claude-code/run-skill-tests.sh@@ -0,0 +1,162 @@+#!/usr/bin/env bash+# Test runner for Claude Code skills+# Tests skills by invoking Claude Code CLI and verifying behavior+set -euo pipefail++SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"+cd "$SCRIPT_DIR"++echo "========================================"+echo " Claude Code Skills Test Suite"+echo "========================================"+echo ""+echo "Repository: $(cd ../.. && pwd)"+echo "Test time: $(date)"+echo "Claude version: $(claude --version 2>/dev/null || echo 'not found')"+echo ""++# Check if Claude Code is available+if ! command -v claude &> /dev/null; then+ echo "ERROR: Claude Code CLI not found"+ echo "Install Claude Code first: https://code.claude.com"+ exit 1+fi++# Parse command line arguments+VERBOSE=false+SPECIFIC_TEST=""+TIMEOUT=300 # Default 5 minute timeout per test++while [[ $# -gt 0 ]]; do+ case $1 in+ --verbose|-v)+ VERBOSE=true+ shift+ ;;+ --test|-t)+ SPECIFIC_TEST="$2"+ shift 2+ ;;+ --timeout)+ TIMEOUT="$2"+ shift 2+ ;;+ --help|-h)+ echo "Usage: $0 [options]"+ echo ""+ echo "Options:"+ echo " --verbose, -v Show verbose output"+ echo " --test, -t NAME Run only the specified test"+ echo " --timeout SECONDS Set timeout per test (default: 300)"+ echo " --help, -h Show this help"+ echo ""+ echo "Tests:"+ echo " test-subagent-driven-development.sh Test subagent-driven-development workflow"+ exit 0+ ;;+ *)+ echo "Unknown option: $1"+ echo "Use --help for usage information"+ exit 1+ ;;+ esac+done++# List of skill tests to run+tests=(+ "test-subagent-driven-development.sh"+)++# Filter to specific test if requested+if [ -n "$SPECIFIC_TEST" ]; then+ tests=("$SPECIFIC_TEST")+fi++# Track results+passed=0+failed=0+skipped=0++# Run each test+for test in "${tests[@]}"; do+ echo "----------------------------------------"+ echo "Running: $test"+ echo "----------------------------------------"++ test_path="$SCRIPT_DIR/$test"++ if [ ! -f "$test_path" ]; then+ echo " [SKIP] Test file not found: $test"+ skipped=$((skipped + 1))+ continue+ fi++ if [ ! -x "$test_path" ]; then+ echo " Making $test executable..."+ chmod +x "$test_path"+ fi++ start_time=$(date +%s)++ if [ "$VERBOSE" = true ]; then+ if timeout "$TIMEOUT" bash "$test_path"; then+ end_time=$(date +%s)+ duration=$((end_time - start_time))+ echo ""+ echo " [PASS] $test (${duration}s)"+ passed=$((passed + 1))+ else+ exit_code=$?+ end_time=$(date +%s)+ duration=$((end_time - start_time))+ echo ""+ if [ $exit_code -eq 124 ]; then+ echo " [FAIL] $test (timeout after ${TIMEOUT}s)"+ else+ echo " [FAIL] $test (${duration}s)"+ fi+ failed=$((failed + 1))+ fi+ else+ # Capture output for non-verbose mode+ if output=$(timeout "$TIMEOUT" bash "$test_path" 2>&1); then+ end_time=$(date +%s)+ duration=$((end_time - start_time))+ echo " [PASS] (${duration}s)"+ passed=$((passed + 1))+ else+ exit_code=$?+ end_time=$(date +%s)+ duration=$((end_time - start_time))+ if [ $exit_code -eq 124 ]; then+ echo " [FAIL] (timeout after ${TIMEOUT}s)"+ else+ echo " [FAIL] (${duration}s)"+ fi+ echo ""+ echo " Output:"+ echo "$output" | sed 's/^/ /'+ failed=$((failed + 1))+ fi+ fi++ echo ""+done++# Print summary+echo "========================================"+echo " Test Results Summary"+echo "========================================"+echo ""+echo " Passed: $passed"+echo " Failed: $failed"+echo " Skipped: $skipped"+echo ""++if [ $failed -gt 0 ]; then+ echo "STATUS: FAILED"+ exit 1+else+ echo "STATUS: PASSED"+ exit 0+fidiff --git a/tests/claude-code/test-helpers.sh b/tests/claude-code/test-helpers.shnew file mode 100755index 000000000..242b16bbd--- /dev/null+++ b/tests/claude-code/test-helpers.sh@@ -0,0 +1,195 @@+#!/usr/bin/env bash+# Helper functions for Claude Code skill tests++# Run Claude Code with a prompt and capture output+# Usage: run_claude "prompt text" [timeout_seconds]+run_claude() {+ local prompt="$1"+ local timeout="${2:-60}"+ local output_file=$(mktemp)++ # Run Claude in headless mode with timeout+ if timeout "$timeout" claude -p "$prompt" > "$output_file" 2>&1; then+ cat "$output_file"+ rm -f "$output_file"+ return 0+ else+ local exit_code=$?+ cat "$output_file" >&2+ rm -f "$output_file"+ return $exit_code+ fi+}++# Check if output contains a pattern+# Usage: assert_contains "output" "pattern" "test name"+assert_contains() {+ local output="$1"+ local pattern="$2"+ local test_name="${3:-test}"++ if echo "$output" | grep -q "$pattern"; then+ echo " [PASS] $test_name"+ return 0+ else+ echo " [FAIL] $test_name"+ echo " Expected to find: $pattern"+ echo " In output:"+ echo "$output" | sed 's/^/ /'+ return 1+ fi+}++# Check if output does NOT contain a pattern+# Usage: assert_not_contains "output" "pattern" "test name"+assert_not_contains() {+ local output="$1"+ local pattern="$2"+ local test_name="${3:-test}"++ if echo "$output" | grep -q "$pattern"; then+ echo " [FAIL] $test_name"+ echo " Did not expect to find: $pattern"+ echo " In output:"+ echo "$output" | sed 's/^/ /'+ return 1+ else+ echo " [PASS] $test_name"+ return 0+ fi+}++# Check if output matches a count+# Usage: assert_count "output" "pattern" expected_count "test name"+assert_count() {+ local output="$1"+ local pattern="$2"+ local expected="$3"+ local test_name="${4:-test}"++ local actual=$(echo "$output" | grep -c "$pattern" || echo "0")++ if [ "$actual" -eq "$expected" ]; then+ echo " [PASS] $test_name (found $actual instances)"+ return 0+ else+ echo " [FAIL] $test_name"+ echo " Expected $expected instances of: $pattern"+ echo " Found $actual instances"+ echo " In output:"+ echo "$output" | sed 's/^/ /'+ return 1+ fi+}++# Check if pattern A appears before pattern B+# Usage: assert_order "output" "pattern_a" "pattern_b" "test name"+assert_order() {+ local output="$1"+ local pattern_a="$2"+ local pattern_b="$3"+ local test_name="${4:-test}"++ # Get line numbers where patterns appear+ local line_a=$(echo "$output" | grep -n "$pattern_a" | head -1 | cut -d: -f1)+ local line_b=$(echo "$output" | grep -n "$pattern_b" | head -1 | cut -d: -f1)++ if [ -z "$line_a" ]; then+ echo " [FAIL] $test_name: pattern A not found: $pattern_a"+ return 1+ fi++ if [ -z "$line_b" ]; then+ echo " [FAIL] $test_name: pattern B not found: $pattern_b"+ return 1+ fi++ if [ "$line_a" -lt "$line_b" ]; then+ echo " [PASS] $test_name (A at line $line_a, B at line $line_b)"+ return 0+ else+ echo " [FAIL] $test_name"+ echo " Expected '$pattern_a' before '$pattern_b'"+ echo " But found A at line $line_a, B at line $line_b"+ return 1+ fi+}++# Create a temporary test project directory+# Usage: test_project=$(create_test_project)+create_test_project() {+ local test_dir=$(mktemp -d)+ echo "$test_dir"+}++# Cleanup test project+# Usage: cleanup_test_project "$test_dir"+cleanup_test_project() {+ local test_dir="$1"+ if [ -d "$test_dir" ]; then+ rm -rf "$test_dir"+ fi+}++# Create a simple plan file for testing+# Usage: create_test_plan "$project_dir" "$plan_name"+create_test_plan() {+ local project_dir="$1"+ local plan_name="${2:-test-plan}"+ local plan_file="$project_dir/docs/plans/$plan_name.md"++ mkdir -p "$(dirname "$plan_file")"++ cat > "$plan_file" <<'EOF'+# Test Implementation Plan++## Task 1: Create Hello Function++Create a simple hello function that returns "Hello, World!".++**File:** `src/hello.js`++**Implementation:**+```javascript+export function hello() {+ return "Hello, World!";+}+```++**Tests:** Write a test that verifies the function returns the expected string.++**Verification:** `npm test`++## Task 2: Create Goodbye Function++Create a goodbye function that takes a name and returns a goodbye message.++**File:** `src/goodbye.js`++**Implementation:**+```javascript+export function goodbye(name) {+ return `Goodbye, ${name}!`;+}+```++**Tests:** Write tests for:+- Default name+- Custom name+- Edge cases (empty string, null)++**Verification:** `npm test`+EOF++ echo "$plan_file"+}++# Export functions for use in tests+export -f run_claude+export -f assert_contains+export -f assert_not_contains+export -f assert_count+export -f assert_order+export -f create_test_project+export -f cleanup_test_project+export -f create_test_plandiff --git a/tests/claude-code/test-subagent-driven-development.sh b/tests/claude-code/test-subagent-driven-development.shnew file mode 100755index 000000000..8edea06fa--- /dev/null+++ b/tests/claude-code/test-subagent-driven-development.sh@@ -0,0 +1,139 @@+#!/usr/bin/env bash+# Test: subagent-driven-development skill+# Verifies that the skill is loaded and follows correct workflow+set -euo pipefail++SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"+source "$SCRIPT_DIR/test-helpers.sh"++echo "=== Test: subagent-driven-development skill ==="+echo ""++# Test 1: Verify skill can be loaded+echo "Test 1: Skill loading..."++output=$(run_claude "What is the subagent-driven-development skill? Describe its key steps briefly." 30)++if assert_contains "$output" "subagent-driven-development" "Skill is recognized"; then+ : # pass+else+ exit 1+fi++if assert_contains "$output" "Load Plan\|read.*plan\|extract.*tasks" "Mentions loading plan"; then+ : # pass+else+ exit 1+fi++echo ""++# Test 2: Verify skill describes correct workflow order+echo "Test 2: Workflow ordering..."++output=$(run_claude "In the subagent-driven-development skill, what comes first: spec compliance review or code quality review? Be specific about the order." 30)++if assert_order "$output" "spec.*compliance" "code.*quality" "Spec compliance before code quality"; then+ : # pass+else+ exit 1+fi++echo ""++# Test 3: Verify self-review is mentioned+echo "Test 3: Self-review requirement..."++output=$(run_claude "Does the subagent-driven-development skill require implementers to do self-review? What should they check?" 30)++if assert_contains "$output" "self-review\|self review" "Mentions self-review"; then+ : # pass+else+ exit 1+fi++if assert_contains "$output" "completeness\|Completeness" "Checks completeness"; then+ : # pass+else+ exit 1+fi++echo ""++# Test 4: Verify plan is read once+echo "Test 4: Plan reading efficiency..."++output=$(run_claude "In subagent-driven-development, how many times should the controller read the plan file? When does this happen?" 30)++if assert_contains "$output" "once\|one time\|single" "Read plan once"; then+ : # pass+else+ exit 1+fi++if assert_contains "$output" "Step 1\|beginning\|start\|Load Plan" "Read at beginning"; then+ : # pass+else+ exit 1+fi++echo ""++# Test 5: Verify spec compliance reviewer is skeptical+echo "Test 5: Spec compliance reviewer mindset..."++output=$(run_claude "What is the spec compliance reviewer's attitude toward the implementer's report in subagent-driven-development?" 30)++if assert_contains "$output" "not trust\|don't trust\|skeptical\|verify.*independently\|suspiciously" "Reviewer is skeptical"; then+ : # pass+else+ exit 1+fi++if assert_contains "$output" "read.*code\|inspect.*code\|verify.*code" "Reviewer reads code"; then+ : # pass+else+ exit 1+fi++echo ""++# Test 6: Verify review loops+echo "Test 6: Review loop requirements..."++output=$(run_claude "In subagent-driven-development, what happens if a reviewer finds issues? Is it a one-time review or a loop?" 30)++if assert_contains "$output" "loop\|again\|repeat\|until.*approved\|until.*compliant" "Review loops mentioned"; then+ : # pass+else+ exit 1+fi++if assert_contains "$output" "implementer.*fix\|fix.*issues" "Implementer fixes issues"; then+ : # pass+else+ exit 1+fi++echo ""++# Test 7: Verify full task text is provided+echo "Test 7: Task context provision..."++output=$(run_claude "In subagent-driven-development, how does the controller provide task information to the implementer subagent? Does it make them read a file or provide it directly?" 30)++if assert_contains "$output" "provide.*directly\|full.*text\|paste\|include.*prompt" "Provides text directly"; then+ : # pass+else+ exit 1+fi++if assert_not_contains "$output" "read.*file\|open.*file" "Doesn't make subagent read file"; then+ : # pass+else+ exit 1+fi++echo ""++echo "=== All subagent-driven-development skill tests passed ==="