Skip to content

An agentic skills framework & software development methodology that works.

Add skill triggering test framework

Jesse Vincentae0ef56b442c
diff --git a/tests/skill-triggering/prompts/dispatching-parallel-agents.txt b/tests/skill-triggering/prompts/dispatching-parallel-agents.txt
new file mode 100644
index 000000000..fb5423f29
--- /dev/null
+++ b/tests/skill-triggering/prompts/dispatching-parallel-agents.txt
@@ -0,0 +1,8 @@
+I have 4 independent test failures happening in different modules:
+
+1. tests/auth/login.test.ts - "should redirect after login" is failing
+2. tests/api/users.test.ts - "should return user list" returns 500
+3. tests/components/Button.test.tsx - snapshot mismatch
+4. tests/utils/date.test.ts - timezone handling broken
+
+These are unrelated issues in different parts of the codebase. Can you investigate all of them?
\ No newline at end of file
diff --git a/tests/skill-triggering/prompts/executing-plans.txt b/tests/skill-triggering/prompts/executing-plans.txt
new file mode 100644
index 000000000..1163636b7
--- /dev/null
+++ b/tests/skill-triggering/prompts/executing-plans.txt
@@ -0,0 +1 @@
+I have a plan document at docs/plans/2024-01-15-auth-system.md that needs to be executed. Please implement it.
\ No newline at end of file
diff --git a/tests/skill-triggering/prompts/requesting-code-review.txt b/tests/skill-triggering/prompts/requesting-code-review.txt
new file mode 100644
index 000000000..f1be2672a
--- /dev/null
+++ b/tests/skill-triggering/prompts/requesting-code-review.txt
@@ -0,0 +1,3 @@
+I just finished implementing the user authentication feature. All the code is committed. Can you review the changes before I merge to main?
+
+The commits are between abc123 and def456.
\ No newline at end of file
diff --git a/tests/skill-triggering/prompts/systematic-debugging.txt b/tests/skill-triggering/prompts/systematic-debugging.txt
new file mode 100644
index 000000000..d3806b9c2
--- /dev/null
+++ b/tests/skill-triggering/prompts/systematic-debugging.txt
@@ -0,0 +1,11 @@
+The tests are failing with this error:
+
+```
+FAIL src/utils/parser.test.ts
+ ● Parser › should handle nested objects
+ TypeError: Cannot read property 'value' of undefined
+ at parse (src/utils/parser.ts:42:18)
+ at Object.<anonymous> (src/utils/parser.test.ts:28:20)
+```
+
+Can you figure out what's going wrong and fix it?
\ No newline at end of file
diff --git a/tests/skill-triggering/prompts/test-driven-development.txt b/tests/skill-triggering/prompts/test-driven-development.txt
new file mode 100644
index 000000000..f386eeab0
--- /dev/null
+++ b/tests/skill-triggering/prompts/test-driven-development.txt
@@ -0,0 +1,7 @@
+I need to add a new feature to validate email addresses. It should:
+- Check that there's an @ symbol
+- Check that there's at least one character before the @
+- Check that there's a dot in the domain part
+- Return true/false
+
+Can you implement this?
\ No newline at end of file
diff --git a/tests/skill-triggering/prompts/writing-plans.txt b/tests/skill-triggering/prompts/writing-plans.txt
new file mode 100644
index 000000000..74803133a
--- /dev/null
+++ b/tests/skill-triggering/prompts/writing-plans.txt
@@ -0,0 +1,10 @@
+Here's the spec for our new authentication system:
+
+Requirements:
+- Users can register with email/password
+- Users can log in and receive a JWT token
+- Protected routes require valid JWT
+- Tokens expire after 24 hours
+- Support password reset via email
+
+We need to implement this. There are multiple steps involved - user model, auth routes, middleware, email service integration.
\ No newline at end of file
diff --git a/tests/skill-triggering/run-all.sh b/tests/skill-triggering/run-all.sh
new file mode 100755
index 000000000..bab5c2dc2
--- /dev/null
+++ b/tests/skill-triggering/run-all.sh
@@ -0,0 +1,60 @@
+#!/bin/bash
+# Run all skill triggering tests
+# Usage: ./run-all.sh
+
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROMPTS_DIR="$SCRIPT_DIR/prompts"
+
+SKILLS=(
+ "systematic-debugging"
+ "test-driven-development"
+ "writing-plans"
+ "dispatching-parallel-agents"
+ "executing-plans"
+ "requesting-code-review"
+)
+
+echo "=== Running Skill Triggering Tests ==="
+echo ""
+
+PASSED=0
+FAILED=0
+RESULTS=()
+
+for skill in "${SKILLS[@]}"; do
+ prompt_file="$PROMPTS_DIR/${skill}.txt"
+
+ if [ ! -f "$prompt_file" ]; then
+ echo "⚠️ SKIP: No prompt file for $skill"
+ continue
+ fi
+
+ echo "Testing: $skill"
+
+ if "$SCRIPT_DIR/run-test.sh" "$skill" "$prompt_file" 3 2>&1 | tee /tmp/skill-test-$skill.log; then
+ PASSED=$((PASSED + 1))
+ RESULTS+=("✅ $skill")
+ else
+ FAILED=$((FAILED + 1))
+ RESULTS+=("❌ $skill")
+ fi
+
+ echo ""
+ echo "---"
+ echo ""
+done
+
+echo ""
+echo "=== Summary ==="
+for result in "${RESULTS[@]}"; do
+ echo " $result"
+done
+echo ""
+echo "Passed: $PASSED"
+echo "Failed: $FAILED"
+
+if [ $FAILED -gt 0 ]; then
+ exit 1
+fi
diff --git a/tests/skill-triggering/run-test.sh b/tests/skill-triggering/run-test.sh
new file mode 100755
index 000000000..553a0e9d6
--- /dev/null
+++ b/tests/skill-triggering/run-test.sh
@@ -0,0 +1,88 @@
+#!/bin/bash
+# Test skill triggering with naive prompts
+# Usage: ./run-test.sh <skill-name> <prompt-file>
+#
+# Tests whether Claude triggers a skill based on a natural prompt
+# (without explicitly mentioning the skill)
+
+set -e
+
+SKILL_NAME="$1"
+PROMPT_FILE="$2"
+MAX_TURNS="${3:-3}"
+
+if [ -z "$SKILL_NAME" ] || [ -z "$PROMPT_FILE" ]; then
+ echo "Usage: $0 <skill-name> <prompt-file> [max-turns]"
+ echo "Example: $0 systematic-debugging ./test-prompts/debugging.txt"
+ exit 1
+fi
+
+# Get the directory where this script lives (should be tests/skill-triggering)
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# Get the superpowers plugin root (two levels up from tests/skill-triggering)
+PLUGIN_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
+
+TIMESTAMP=$(date +%s)
+OUTPUT_DIR="/tmp/superpowers-tests/${TIMESTAMP}/skill-triggering/${SKILL_NAME}"
+mkdir -p "$OUTPUT_DIR"
+
+# Read prompt from file
+PROMPT=$(cat "$PROMPT_FILE")
+
+echo "=== Skill Triggering Test ==="
+echo "Skill: $SKILL_NAME"
+echo "Prompt file: $PROMPT_FILE"
+echo "Max turns: $MAX_TURNS"
+echo "Output dir: $OUTPUT_DIR"
+echo ""
+
+# Copy prompt for reference
+cp "$PROMPT_FILE" "$OUTPUT_DIR/prompt.txt"
+
+# Run Claude
+LOG_FILE="$OUTPUT_DIR/claude-output.json"
+cd "$OUTPUT_DIR"
+
+echo "Plugin dir: $PLUGIN_DIR"
+echo "Running claude -p with naive prompt..."
+timeout 300 claude -p "$PROMPT" \
+ --plugin-dir "$PLUGIN_DIR" \
+ --dangerously-skip-permissions \
+ --max-turns "$MAX_TURNS" \
+ --output-format stream-json \
+ > "$LOG_FILE" 2>&1 || true
+
+echo ""
+echo "=== Results ==="
+
+# Check if skill was triggered (look for Skill tool invocation)
+# In stream-json, tool invocations have "name":"Skill" (not "tool":"Skill")
+# Match either "skill":"skillname" or "skill":"namespace:skillname"
+SKILL_PATTERN='"skill":"([^"]*:)?'"${SKILL_NAME}"'"'
+if grep -q '"name":"Skill"' "$LOG_FILE" && grep -qE "$SKILL_PATTERN" "$LOG_FILE"; then
+ echo "✅ PASS: Skill '$SKILL_NAME' was triggered"
+ TRIGGERED=true
+else
+ echo "❌ FAIL: Skill '$SKILL_NAME' was NOT triggered"
+ TRIGGERED=false
+fi
+
+# Show what skills WERE triggered
+echo ""
+echo "Skills triggered in this run:"
+grep -o '"skill":"[^"]*"' "$LOG_FILE" 2>/dev/null | sort -u || echo " (none)"
+
+# Show first assistant message
+echo ""
+echo "First assistant response (truncated):"
+grep '"type":"assistant"' "$LOG_FILE" | head -1 | jq -r '.message.content[0].text // .message.content' 2>/dev/null | head -c 500 || echo " (could not extract)"
+
+echo ""
+echo "Full log: $LOG_FILE"
+echo "Timestamp: $TIMESTAMP"
+
+if [ "$TRIGGERED" = "true" ]; then
+ exit 0
+else
+ exit 1
+fi