Skip to content

An agentic skills framework & software development methodology that works.

Add Claude Code skills test framework

Jesse Vincent51a171cd141f
diff --git a/tests/claude-code/README.md b/tests/claude-code/README.md
new file mode 100644
index 000000000..177f791a1
--- /dev/null
+++ b/tests/claude-code/README.md
@@ -0,0 +1,128 @@
+# Claude Code Skills Tests
+
+Automated tests for superpowers skills using Claude Code CLI.
+
+## Overview
+
+This test suite verifies that skills are loaded correctly and Claude follows them as expected. Tests invoke Claude Code in headless mode (`claude -p`) and verify the behavior.
+
+## Requirements
+
+- Claude Code CLI installed and in PATH (`claude --version` should work)
+- Local superpowers plugin installed (see main README for installation)
+
+## Running Tests
+
+### Run all tests:
+```bash
+./run-skill-tests.sh
+```
+
+### Run specific test:
+```bash
+./run-skill-tests.sh --test test-subagent-driven-development.sh
+```
+
+### Run with verbose output:
+```bash
+./run-skill-tests.sh --verbose
+```
+
+### Set custom timeout (default 300s):
+```bash
+./run-skill-tests.sh --timeout 600
+```
+
+## Test Structure
+
+### test-helpers.sh
+Common functions for skills testing:
+- `run_claude "prompt" [timeout]` - Run Claude with prompt
+- `assert_contains output pattern name` - Verify pattern exists
+- `assert_not_contains output pattern name` - Verify pattern absent
+- `assert_count output pattern count name` - Verify exact count
+- `assert_order output pattern_a pattern_b name` - Verify order
+- `create_test_project` - Create temp test directory
+- `create_test_plan project_dir` - Create sample plan file
+
+### Test Files
+
+Each test file:
+1. Sources `test-helpers.sh`
+2. Runs Claude Code with specific prompts
+3. Verifies expected behavior using assertions
+4. Returns 0 on success, non-zero on failure
+
+## Example Test
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/test-helpers.sh"
+
+echo "=== Test: My Skill ==="
+
+# Ask Claude about the skill
+output=$(run_claude "What does the my-skill skill do?" 30)
+
+# Verify response
+assert_contains "$output" "expected behavior" "Skill describes behavior"
+
+echo "=== All tests passed ==="
+```
+
+## Current Tests
+
+### test-subagent-driven-development.sh
+Tests the updated subagent-driven-development workflow:
+- Skill loading
+- Workflow ordering (spec compliance before code quality)
+- Self-review requirements
+- Plan reading efficiency (read once)
+- Spec compliance reviewer skepticism
+- Review loops
+- Task context provision (full text, not file reading)
+
+## Adding New Tests
+
+1. Create new test file: `test-<skill-name>.sh`
+2. Source test-helpers.sh
+3. Write tests using `run_claude` and assertions
+4. Add to test list in `run-skill-tests.sh`
+5. Make executable: `chmod +x test-<skill-name>.sh`
+
+## Timeout Considerations
+
+- Default timeout: 5 minutes per test
+- Claude Code may take time to respond
+- Adjust with `--timeout` if needed
+- Tests should be focused to avoid long runs
+
+## Debugging Failed Tests
+
+With `--verbose`, you'll see full Claude output:
+```bash
+./run-skill-tests.sh --verbose --test test-subagent-driven-development.sh
+```
+
+Without verbose, only failures show output.
+
+## CI/CD Integration
+
+To run in CI:
+```bash
+# Run with explicit timeout for CI environments
+./run-skill-tests.sh --timeout 900
+
+# Exit code 0 = success, non-zero = failure
+```
+
+## Notes
+
+- Tests verify skill *instructions*, not full execution
+- Full workflow tests would be very slow
+- Focus on verifying key skill requirements
+- Tests should be deterministic
+- Avoid testing implementation details
diff --git a/tests/claude-code/run-skill-tests.sh b/tests/claude-code/run-skill-tests.sh
new file mode 100755
index 000000000..a50d71d69
--- /dev/null
+++ b/tests/claude-code/run-skill-tests.sh
@@ -0,0 +1,162 @@
+#!/usr/bin/env bash
+# Test runner for Claude Code skills
+# Tests skills by invoking Claude Code CLI and verifying behavior
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+cd "$SCRIPT_DIR"
+
+echo "========================================"
+echo " Claude Code Skills Test Suite"
+echo "========================================"
+echo ""
+echo "Repository: $(cd ../.. && pwd)"
+echo "Test time: $(date)"
+echo "Claude version: $(claude --version 2>/dev/null || echo 'not found')"
+echo ""
+
+# Check if Claude Code is available
+if ! command -v claude &> /dev/null; then
+ echo "ERROR: Claude Code CLI not found"
+ echo "Install Claude Code first: https://code.claude.com"
+ exit 1
+fi
+
+# Parse command line arguments
+VERBOSE=false
+SPECIFIC_TEST=""
+TIMEOUT=300 # Default 5 minute timeout per test
+
+while [[ $# -gt 0 ]]; do
+ case $1 in
+ --verbose|-v)
+ VERBOSE=true
+ shift
+ ;;
+ --test|-t)
+ SPECIFIC_TEST="$2"
+ shift 2
+ ;;
+ --timeout)
+ TIMEOUT="$2"
+ shift 2
+ ;;
+ --help|-h)
+ echo "Usage: $0 [options]"
+ echo ""
+ echo "Options:"
+ echo " --verbose, -v Show verbose output"
+ echo " --test, -t NAME Run only the specified test"
+ echo " --timeout SECONDS Set timeout per test (default: 300)"
+ echo " --help, -h Show this help"
+ echo ""
+ echo "Tests:"
+ echo " test-subagent-driven-development.sh Test subagent-driven-development workflow"
+ exit 0
+ ;;
+ *)
+ echo "Unknown option: $1"
+ echo "Use --help for usage information"
+ exit 1
+ ;;
+ esac
+done
+
+# List of skill tests to run
+tests=(
+ "test-subagent-driven-development.sh"
+)
+
+# Filter to specific test if requested
+if [ -n "$SPECIFIC_TEST" ]; then
+ tests=("$SPECIFIC_TEST")
+fi
+
+# Track results
+passed=0
+failed=0
+skipped=0
+
+# Run each test
+for test in "${tests[@]}"; do
+ echo "----------------------------------------"
+ echo "Running: $test"
+ echo "----------------------------------------"
+
+ test_path="$SCRIPT_DIR/$test"
+
+ if [ ! -f "$test_path" ]; then
+ echo " [SKIP] Test file not found: $test"
+ skipped=$((skipped + 1))
+ continue
+ fi
+
+ if [ ! -x "$test_path" ]; then
+ echo " Making $test executable..."
+ chmod +x "$test_path"
+ fi
+
+ start_time=$(date +%s)
+
+ if [ "$VERBOSE" = true ]; then
+ if timeout "$TIMEOUT" bash "$test_path"; then
+ end_time=$(date +%s)
+ duration=$((end_time - start_time))
+ echo ""
+ echo " [PASS] $test (${duration}s)"
+ passed=$((passed + 1))
+ else
+ exit_code=$?
+ end_time=$(date +%s)
+ duration=$((end_time - start_time))
+ echo ""
+ if [ $exit_code -eq 124 ]; then
+ echo " [FAIL] $test (timeout after ${TIMEOUT}s)"
+ else
+ echo " [FAIL] $test (${duration}s)"
+ fi
+ failed=$((failed + 1))
+ fi
+ else
+ # Capture output for non-verbose mode
+ if output=$(timeout "$TIMEOUT" bash "$test_path" 2>&1); then
+ end_time=$(date +%s)
+ duration=$((end_time - start_time))
+ echo " [PASS] (${duration}s)"
+ passed=$((passed + 1))
+ else
+ exit_code=$?
+ end_time=$(date +%s)
+ duration=$((end_time - start_time))
+ if [ $exit_code -eq 124 ]; then
+ echo " [FAIL] (timeout after ${TIMEOUT}s)"
+ else
+ echo " [FAIL] (${duration}s)"
+ fi
+ echo ""
+ echo " Output:"
+ echo "$output" | sed 's/^/ /'
+ failed=$((failed + 1))
+ fi
+ fi
+
+ echo ""
+done
+
+# Print summary
+echo "========================================"
+echo " Test Results Summary"
+echo "========================================"
+echo ""
+echo " Passed: $passed"
+echo " Failed: $failed"
+echo " Skipped: $skipped"
+echo ""
+
+if [ $failed -gt 0 ]; then
+ echo "STATUS: FAILED"
+ exit 1
+else
+ echo "STATUS: PASSED"
+ exit 0
+fi
diff --git a/tests/claude-code/test-helpers.sh b/tests/claude-code/test-helpers.sh
new file mode 100755
index 000000000..242b16bbd
--- /dev/null
+++ b/tests/claude-code/test-helpers.sh
@@ -0,0 +1,195 @@
+#!/usr/bin/env bash
+# Helper functions for Claude Code skill tests
+
+# Run Claude Code with a prompt and capture output
+# Usage: run_claude "prompt text" [timeout_seconds]
+run_claude() {
+ local prompt="$1"
+ local timeout="${2:-60}"
+ local output_file=$(mktemp)
+
+ # Run Claude in headless mode with timeout
+ if timeout "$timeout" claude -p "$prompt" > "$output_file" 2>&1; then
+ cat "$output_file"
+ rm -f "$output_file"
+ return 0
+ else
+ local exit_code=$?
+ cat "$output_file" >&2
+ rm -f "$output_file"
+ return $exit_code
+ fi
+}
+
+# Check if output contains a pattern
+# Usage: assert_contains "output" "pattern" "test name"
+assert_contains() {
+ local output="$1"
+ local pattern="$2"
+ local test_name="${3:-test}"
+
+ if echo "$output" | grep -q "$pattern"; then
+ echo " [PASS] $test_name"
+ return 0
+ else
+ echo " [FAIL] $test_name"
+ echo " Expected to find: $pattern"
+ echo " In output:"
+ echo "$output" | sed 's/^/ /'
+ return 1
+ fi
+}
+
+# Check if output does NOT contain a pattern
+# Usage: assert_not_contains "output" "pattern" "test name"
+assert_not_contains() {
+ local output="$1"
+ local pattern="$2"
+ local test_name="${3:-test}"
+
+ if echo "$output" | grep -q "$pattern"; then
+ echo " [FAIL] $test_name"
+ echo " Did not expect to find: $pattern"
+ echo " In output:"
+ echo "$output" | sed 's/^/ /'
+ return 1
+ else
+ echo " [PASS] $test_name"
+ return 0
+ fi
+}
+
+# Check if output matches a count
+# Usage: assert_count "output" "pattern" expected_count "test name"
+assert_count() {
+ local output="$1"
+ local pattern="$2"
+ local expected="$3"
+ local test_name="${4:-test}"
+
+ local actual=$(echo "$output" | grep -c "$pattern" || echo "0")
+
+ if [ "$actual" -eq "$expected" ]; then
+ echo " [PASS] $test_name (found $actual instances)"
+ return 0
+ else
+ echo " [FAIL] $test_name"
+ echo " Expected $expected instances of: $pattern"
+ echo " Found $actual instances"
+ echo " In output:"
+ echo "$output" | sed 's/^/ /'
+ return 1
+ fi
+}
+
+# Check if pattern A appears before pattern B
+# Usage: assert_order "output" "pattern_a" "pattern_b" "test name"
+assert_order() {
+ local output="$1"
+ local pattern_a="$2"
+ local pattern_b="$3"
+ local test_name="${4:-test}"
+
+ # Get line numbers where patterns appear
+ local line_a=$(echo "$output" | grep -n "$pattern_a" | head -1 | cut -d: -f1)
+ local line_b=$(echo "$output" | grep -n "$pattern_b" | head -1 | cut -d: -f1)
+
+ if [ -z "$line_a" ]; then
+ echo " [FAIL] $test_name: pattern A not found: $pattern_a"
+ return 1
+ fi
+
+ if [ -z "$line_b" ]; then
+ echo " [FAIL] $test_name: pattern B not found: $pattern_b"
+ return 1
+ fi
+
+ if [ "$line_a" -lt "$line_b" ]; then
+ echo " [PASS] $test_name (A at line $line_a, B at line $line_b)"
+ return 0
+ else
+ echo " [FAIL] $test_name"
+ echo " Expected '$pattern_a' before '$pattern_b'"
+ echo " But found A at line $line_a, B at line $line_b"
+ return 1
+ fi
+}
+
+# Create a temporary test project directory
+# Usage: test_project=$(create_test_project)
+create_test_project() {
+ local test_dir=$(mktemp -d)
+ echo "$test_dir"
+}
+
+# Cleanup test project
+# Usage: cleanup_test_project "$test_dir"
+cleanup_test_project() {
+ local test_dir="$1"
+ if [ -d "$test_dir" ]; then
+ rm -rf "$test_dir"
+ fi
+}
+
+# Create a simple plan file for testing
+# Usage: create_test_plan "$project_dir" "$plan_name"
+create_test_plan() {
+ local project_dir="$1"
+ local plan_name="${2:-test-plan}"
+ local plan_file="$project_dir/docs/plans/$plan_name.md"
+
+ mkdir -p "$(dirname "$plan_file")"
+
+ cat > "$plan_file" <<'EOF'
+# Test Implementation Plan
+
+## Task 1: Create Hello Function
+
+Create a simple hello function that returns "Hello, World!".
+
+**File:** `src/hello.js`
+
+**Implementation:**
+```javascript
+export function hello() {
+ return "Hello, World!";
+}
+```
+
+**Tests:** Write a test that verifies the function returns the expected string.
+
+**Verification:** `npm test`
+
+## Task 2: Create Goodbye Function
+
+Create a goodbye function that takes a name and returns a goodbye message.
+
+**File:** `src/goodbye.js`
+
+**Implementation:**
+```javascript
+export function goodbye(name) {
+ return `Goodbye, ${name}!`;
+}
+```
+
+**Tests:** Write tests for:
+- Default name
+- Custom name
+- Edge cases (empty string, null)
+
+**Verification:** `npm test`
+EOF
+
+ echo "$plan_file"
+}
+
+# Export functions for use in tests
+export -f run_claude
+export -f assert_contains
+export -f assert_not_contains
+export -f assert_count
+export -f assert_order
+export -f create_test_project
+export -f cleanup_test_project
+export -f create_test_plan
diff --git a/tests/claude-code/test-subagent-driven-development.sh b/tests/claude-code/test-subagent-driven-development.sh
new file mode 100755
index 000000000..8edea06fa
--- /dev/null
+++ b/tests/claude-code/test-subagent-driven-development.sh
@@ -0,0 +1,139 @@
+#!/usr/bin/env bash
+# Test: subagent-driven-development skill
+# Verifies that the skill is loaded and follows correct workflow
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/test-helpers.sh"
+
+echo "=== Test: subagent-driven-development skill ==="
+echo ""
+
+# Test 1: Verify skill can be loaded
+echo "Test 1: Skill loading..."
+
+output=$(run_claude "What is the subagent-driven-development skill? Describe its key steps briefly." 30)
+
+if assert_contains "$output" "subagent-driven-development" "Skill is recognized"; then
+ : # pass
+else
+ exit 1
+fi
+
+if assert_contains "$output" "Load Plan\|read.*plan\|extract.*tasks" "Mentions loading plan"; then
+ : # pass
+else
+ exit 1
+fi
+
+echo ""
+
+# Test 2: Verify skill describes correct workflow order
+echo "Test 2: Workflow ordering..."
+
+output=$(run_claude "In the subagent-driven-development skill, what comes first: spec compliance review or code quality review? Be specific about the order." 30)
+
+if assert_order "$output" "spec.*compliance" "code.*quality" "Spec compliance before code quality"; then
+ : # pass
+else
+ exit 1
+fi
+
+echo ""
+
+# Test 3: Verify self-review is mentioned
+echo "Test 3: Self-review requirement..."
+
+output=$(run_claude "Does the subagent-driven-development skill require implementers to do self-review? What should they check?" 30)
+
+if assert_contains "$output" "self-review\|self review" "Mentions self-review"; then
+ : # pass
+else
+ exit 1
+fi
+
+if assert_contains "$output" "completeness\|Completeness" "Checks completeness"; then
+ : # pass
+else
+ exit 1
+fi
+
+echo ""
+
+# Test 4: Verify plan is read once
+echo "Test 4: Plan reading efficiency..."
+
+output=$(run_claude "In subagent-driven-development, how many times should the controller read the plan file? When does this happen?" 30)
+
+if assert_contains "$output" "once\|one time\|single" "Read plan once"; then
+ : # pass
+else
+ exit 1
+fi
+
+if assert_contains "$output" "Step 1\|beginning\|start\|Load Plan" "Read at beginning"; then
+ : # pass
+else
+ exit 1
+fi
+
+echo ""
+
+# Test 5: Verify spec compliance reviewer is skeptical
+echo "Test 5: Spec compliance reviewer mindset..."
+
+output=$(run_claude "What is the spec compliance reviewer's attitude toward the implementer's report in subagent-driven-development?" 30)
+
+if assert_contains "$output" "not trust\|don't trust\|skeptical\|verify.*independently\|suspiciously" "Reviewer is skeptical"; then
+ : # pass
+else
+ exit 1
+fi
+
+if assert_contains "$output" "read.*code\|inspect.*code\|verify.*code" "Reviewer reads code"; then
+ : # pass
+else
+ exit 1
+fi
+
+echo ""
+
+# Test 6: Verify review loops
+echo "Test 6: Review loop requirements..."
+
+output=$(run_claude "In subagent-driven-development, what happens if a reviewer finds issues? Is it a one-time review or a loop?" 30)
+
+if assert_contains "$output" "loop\|again\|repeat\|until.*approved\|until.*compliant" "Review loops mentioned"; then
+ : # pass
+else
+ exit 1
+fi
+
+if assert_contains "$output" "implementer.*fix\|fix.*issues" "Implementer fixes issues"; then
+ : # pass
+else
+ exit 1
+fi
+
+echo ""
+
+# Test 7: Verify full task text is provided
+echo "Test 7: Task context provision..."
+
+output=$(run_claude "In subagent-driven-development, how does the controller provide task information to the implementer subagent? Does it make them read a file or provide it directly?" 30)
+
+if assert_contains "$output" "provide.*directly\|full.*text\|paste\|include.*prompt" "Provides text directly"; then
+ : # pass
+else
+ exit 1
+fi
+
+if assert_not_contains "$output" "read.*file\|open.*file" "Doesn't make subagent read file"; then
+ : # pass
+else
+ exit 1
+fi
+
+echo ""
+
+echo "=== All subagent-driven-development skill tests passed ==="