An agentic skills framework & software development methodology that works.
Add skill triggering test framework
Jesse Vincentae0ef56b442c
diff --git a/tests/skill-triggering/prompts/dispatching-parallel-agents.txt b/tests/skill-triggering/prompts/dispatching-parallel-agents.txtnew file mode 100644index 000000000..fb5423f29--- /dev/null+++ b/tests/skill-triggering/prompts/dispatching-parallel-agents.txt@@ -0,0 +1,8 @@+I have 4 independent test failures happening in different modules:++1. tests/auth/login.test.ts - "should redirect after login" is failing+2. tests/api/users.test.ts - "should return user list" returns 500+3. tests/components/Button.test.tsx - snapshot mismatch+4. tests/utils/date.test.ts - timezone handling broken++These are unrelated issues in different parts of the codebase. Can you investigate all of them?\ No newline at end of filediff --git a/tests/skill-triggering/prompts/executing-plans.txt b/tests/skill-triggering/prompts/executing-plans.txtnew file mode 100644index 000000000..1163636b7--- /dev/null+++ b/tests/skill-triggering/prompts/executing-plans.txt@@ -0,0 +1 @@+I have a plan document at docs/plans/2024-01-15-auth-system.md that needs to be executed. Please implement it.\ No newline at end of filediff --git a/tests/skill-triggering/prompts/requesting-code-review.txt b/tests/skill-triggering/prompts/requesting-code-review.txtnew file mode 100644index 000000000..f1be2672a--- /dev/null+++ b/tests/skill-triggering/prompts/requesting-code-review.txt@@ -0,0 +1,3 @@+I just finished implementing the user authentication feature. All the code is committed. Can you review the changes before I merge to main?++The commits are between abc123 and def456.\ No newline at end of filediff --git a/tests/skill-triggering/prompts/systematic-debugging.txt b/tests/skill-triggering/prompts/systematic-debugging.txtnew file mode 100644index 000000000..d3806b9c2--- /dev/null+++ b/tests/skill-triggering/prompts/systematic-debugging.txt@@ -0,0 +1,11 @@+The tests are failing with this error:++```+FAIL src/utils/parser.test.ts+ ● Parser › should handle nested objects+ TypeError: Cannot read property 'value' of undefined+ at parse (src/utils/parser.ts:42:18)+ at Object.<anonymous> (src/utils/parser.test.ts:28:20)+```++Can you figure out what's going wrong and fix it?\ No newline at end of filediff --git a/tests/skill-triggering/prompts/test-driven-development.txt b/tests/skill-triggering/prompts/test-driven-development.txtnew file mode 100644index 000000000..f386eeab0--- /dev/null+++ b/tests/skill-triggering/prompts/test-driven-development.txt@@ -0,0 +1,7 @@+I need to add a new feature to validate email addresses. It should:+- Check that there's an @ symbol+- Check that there's at least one character before the @+- Check that there's a dot in the domain part+- Return true/false++Can you implement this?\ No newline at end of filediff --git a/tests/skill-triggering/prompts/writing-plans.txt b/tests/skill-triggering/prompts/writing-plans.txtnew file mode 100644index 000000000..74803133a--- /dev/null+++ b/tests/skill-triggering/prompts/writing-plans.txt@@ -0,0 +1,10 @@+Here's the spec for our new authentication system:++Requirements:+- Users can register with email/password+- Users can log in and receive a JWT token+- Protected routes require valid JWT+- Tokens expire after 24 hours+- Support password reset via email++We need to implement this. There are multiple steps involved - user model, auth routes, middleware, email service integration.\ No newline at end of filediff --git a/tests/skill-triggering/run-all.sh b/tests/skill-triggering/run-all.shnew file mode 100755index 000000000..bab5c2dc2--- /dev/null+++ b/tests/skill-triggering/run-all.sh@@ -0,0 +1,60 @@+#!/bin/bash+# Run all skill triggering tests+# Usage: ./run-all.sh++set -e++SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+PROMPTS_DIR="$SCRIPT_DIR/prompts"++SKILLS=(+ "systematic-debugging"+ "test-driven-development"+ "writing-plans"+ "dispatching-parallel-agents"+ "executing-plans"+ "requesting-code-review"+)++echo "=== Running Skill Triggering Tests ==="+echo ""++PASSED=0+FAILED=0+RESULTS=()++for skill in "${SKILLS[@]}"; do+ prompt_file="$PROMPTS_DIR/${skill}.txt"++ if [ ! -f "$prompt_file" ]; then+ echo "⚠️ SKIP: No prompt file for $skill"+ continue+ fi++ echo "Testing: $skill"++ if "$SCRIPT_DIR/run-test.sh" "$skill" "$prompt_file" 3 2>&1 | tee /tmp/skill-test-$skill.log; then+ PASSED=$((PASSED + 1))+ RESULTS+=("✅ $skill")+ else+ FAILED=$((FAILED + 1))+ RESULTS+=("❌ $skill")+ fi++ echo ""+ echo "---"+ echo ""+done++echo ""+echo "=== Summary ==="+for result in "${RESULTS[@]}"; do+ echo " $result"+done+echo ""+echo "Passed: $PASSED"+echo "Failed: $FAILED"++if [ $FAILED -gt 0 ]; then+ exit 1+fidiff --git a/tests/skill-triggering/run-test.sh b/tests/skill-triggering/run-test.shnew file mode 100755index 000000000..553a0e9d6--- /dev/null+++ b/tests/skill-triggering/run-test.sh@@ -0,0 +1,88 @@+#!/bin/bash+# Test skill triggering with naive prompts+# Usage: ./run-test.sh <skill-name> <prompt-file>+#+# Tests whether Claude triggers a skill based on a natural prompt+# (without explicitly mentioning the skill)++set -e++SKILL_NAME="$1"+PROMPT_FILE="$2"+MAX_TURNS="${3:-3}"++if [ -z "$SKILL_NAME" ] || [ -z "$PROMPT_FILE" ]; then+ echo "Usage: $0 <skill-name> <prompt-file> [max-turns]"+ echo "Example: $0 systematic-debugging ./test-prompts/debugging.txt"+ exit 1+fi++# Get the directory where this script lives (should be tests/skill-triggering)+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+# Get the superpowers plugin root (two levels up from tests/skill-triggering)+PLUGIN_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"++TIMESTAMP=$(date +%s)+OUTPUT_DIR="/tmp/superpowers-tests/${TIMESTAMP}/skill-triggering/${SKILL_NAME}"+mkdir -p "$OUTPUT_DIR"++# Read prompt from file+PROMPT=$(cat "$PROMPT_FILE")++echo "=== Skill Triggering Test ==="+echo "Skill: $SKILL_NAME"+echo "Prompt file: $PROMPT_FILE"+echo "Max turns: $MAX_TURNS"+echo "Output dir: $OUTPUT_DIR"+echo ""++# Copy prompt for reference+cp "$PROMPT_FILE" "$OUTPUT_DIR/prompt.txt"++# Run Claude+LOG_FILE="$OUTPUT_DIR/claude-output.json"+cd "$OUTPUT_DIR"++echo "Plugin dir: $PLUGIN_DIR"+echo "Running claude -p with naive prompt..."+timeout 300 claude -p "$PROMPT" \+ --plugin-dir "$PLUGIN_DIR" \+ --dangerously-skip-permissions \+ --max-turns "$MAX_TURNS" \+ --output-format stream-json \+ > "$LOG_FILE" 2>&1 || true++echo ""+echo "=== Results ==="++# Check if skill was triggered (look for Skill tool invocation)+# In stream-json, tool invocations have "name":"Skill" (not "tool":"Skill")+# Match either "skill":"skillname" or "skill":"namespace:skillname"+SKILL_PATTERN='"skill":"([^"]*:)?'"${SKILL_NAME}"'"'+if grep -q '"name":"Skill"' "$LOG_FILE" && grep -qE "$SKILL_PATTERN" "$LOG_FILE"; then+ echo "✅ PASS: Skill '$SKILL_NAME' was triggered"+ TRIGGERED=true+else+ echo "❌ FAIL: Skill '$SKILL_NAME' was NOT triggered"+ TRIGGERED=false+fi++# Show what skills WERE triggered+echo ""+echo "Skills triggered in this run:"+grep -o '"skill":"[^"]*"' "$LOG_FILE" 2>/dev/null | sort -u || echo " (none)"++# Show first assistant message+echo ""+echo "First assistant response (truncated):"+grep '"type":"assistant"' "$LOG_FILE" | head -1 | jq -r '.message.content[0].text // .message.content' 2>/dev/null | head -c 500 || echo " (could not extract)"++echo ""+echo "Full log: $LOG_FILE"+echo "Timestamp: $TIMESTAMP"++if [ "$TRIGGERED" = "true" ]; then+ exit 0+else+ exit 1+fi