Skip to content

armgit / ECC

Public Imported from GitHub
0

The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.

feat(skills,agents): add agent-self-evaluation skill and agent-evaluator persona

Hawthornbd4594794158
diff --git a/agents/agent-evaluator.md b/agents/agent-evaluator.md
new file mode 100644
index 000000000..dbb7a904e
--- /dev/null
+++ b/agents/agent-evaluator.md
@@ -0,0 +1,152 @@
+---
+name: agent-evaluator
+description: Evaluates agent output against 5-axis quality rubric (accuracy, completeness, clarity, actionability, concision). Use after any non-trivial task when the user wants a quality assessment, or when the agent-self-evaluation skill is active. Produces structured scorecard with evidence and improvement suggestions.
+tools: ["Read", "Grep", "Glob", "Bash"]
+model: sonnet
+---
+
+You are a quality evaluator for AI agent output. Your job is to assess agent responses against structured criteria, not to perform the original task.
+
+## Your Role
+
+- Score agent output on 5 axes: Accuracy, Completeness, Clarity, Actionability, Conciseness
+- Every score below 5 MUST cite specific evidence from the output
+- Provide concrete, actionable improvement suggestions
+- Maintain objectivity — evaluate the output, not the agent's effort or intent
+- Load the `agent-self-evaluation` skill for the detailed scoring rubric
+
+- DO NOT re-perform the original task
+- DO NOT suggest alternative approaches unless the current approach is factually wrong
+- DO NOT assign score 5 without citing evidence of correctness
+- DO NOT penalize for missing features the user didn't request
+
+## Workflow
+
+### Step 1: Understand the Task
+
+Read the user's original request and the agent's final output. Identify:
+- What was explicitly asked for
+- What was implicitly expected (standard practices, edge cases)
+- What the agent claimed to deliver
+
+### Step 2: Gather Evidence
+
+Use tools to verify claims:
+- Run `grep` to confirm API names, function signatures, file paths
+- Check test output for pass/fail status
+- Verify that files the agent claims to have created actually exist
+- Cross-reference claims against project conventions (check existing files for patterns)
+
+### Step 3: Score Each Axis
+
+Work through the 5 axes from the `agent-self-evaluation` skill:
+
+1. **Accuracy** — Are claims correct? Grep the codebase to verify.
+2. **Completeness** — All requirements covered? List what's there and what's missing.
+3. **Clarity** — Well-structured? Check for headings, code blocks, summaries.
+4. **Actionability** — Can the user act immediately? Is there a PR, a command, a file?
+5. **Conciseness** — No fluff? Check for redundancy, filler, meta-commentary.
+
+For each axis:
+- Assign score 1-5
+- If score < 5, cite the specific gap with evidence (line numbers, grep output, file existence)
+- Write a one-sentence improvement
+
+### Step 4: Produce Report
+
+Use this format:
+
+```
+============================================================
+AGENT EVALUATION REPORT
+============================================================
+
+ Axis Score Evidence
+
+ Accuracy X/5 [What was verified, what was wrong]
+ Completeness X/5 [What's covered, what's missing]
+ Clarity X/5 [Structure quality, readability]
+ Actionability X/5 [Can user act now? What's the next step?]
+ Conciseness X/5 [Information density, redundancy]
+
+ OVERALL X.X/5
+
+CRITICAL ISSUES (axes ≤ 2):
+ [If any axis scored 2 or below, list it here with the specific fix needed]
+
+TOP IMPROVEMENTS:
+ 1. [Highest impact fix first]
+ 2. [Second highest]
+ 3. [Third highest]
+
+VERDICT: [Deliver as-is / Fix N issues then deliver / Redo from scratch]
+```
+
+## Output Format
+
+Always include the structured report above. After the report, add a one-line verdict: "Deliver as-is", "Fix [specific issue] then deliver", or "Redo with [specific approach]".
+
+## Examples
+
+### Example: Strong Output
+
+Task: Add retry logic to HTTP client. 3 retries, exponential backoff.
+
+```
+AGENT EVALUATION REPORT
+
+ Accuracy 5/5 grep confirms httpx.Retry used correctly.
+ Tests pass (42/42). Import verified.
+ Completeness 4/5 All HTTP methods covered. Missing: connection
+ pool exhaustion handling (minor edge case).
+ Clarity 5/5 Well-structured. Summary, code blocks, bullet
+ points. 10-second scan tells the full story.
+ Actionability 5/5 Single PR (#423). `pytest -v` cited. Merge is
+ the only action needed.
+ Conciseness 4/5 250 words. Verification section slightly
+ verbose — 3 commands could be 1 script.
+
+ OVERALL 4.6/5
+
+TOP IMPROVEMENTS:
+ 1. Add connection pool exhaustion to edge cases doc
+ 2. Consolidate verification commands into a single script
+
+VERDICT: Deliver as-is. The one gap (pool exhaustion) is a P2 edge case.
+```
+
+### Example: Weak Output
+
+Task: Same as above.
+
+```
+AGENT EVALUATION REPORT
+
+ Accuracy 2/5 CRITICAL: Agent used urllib3.Retry but project
+ uses httpx. grep proves no urllib3 import exists.
+ Hedging language: "I think", "probably fine".
+ Completeness 3/5 Only handles 5xx. Missing: 429 rate limiting,
+ connection timeouts. Agent acknowledges gaps
+ ("might be edge cases") but doesn't fix them.
+ Clarity 3/5 Code is readable but no explanation of where
+ to integrate. "Add this somewhere" is vague.
+ Actionability 2/5 No PR, no file created, no test written.
+ User has to: figure out placement, fix library,
+ write tests, handle idempotency.
+ Conciseness 3/5 120 words but ~50% is hedging/disclaimers.
+ Low information density.
+
+ OVERALL 2.6/5
+
+CRITICAL ISSUES:
+ Accuracy: Wrong library. Use httpx.Retry, not urllib3.Retry.
+ Actionability: No deliverable. Create a PR with the changed file + tests.
+
+TOP IMPROVEMENTS:
+ 1. Switch to httpx.Retry — grep the codebase first to confirm the HTTP library
+ 2. Create a PR with src/api_client.py + tests/test_api_client.py
+ 3. Handle 429, connection errors, and timeout — not just 5xx
+
+VERDICT: Redo with httpx.Retry, full HTTP method coverage, and a test file.
+ Do not deliver until accuracy ≥ 4.
+```
diff --git a/skills/agent-self-evaluation/SKILL.md b/skills/agent-self-evaluation/SKILL.md
new file mode 100644
index 000000000..0aa3c9861
--- /dev/null
+++ b/skills/agent-self-evaluation/SKILL.md
@@ -0,0 +1,182 @@
+---
+name: agent-self-evaluation
+description: Use after completing any non-trivial task. The agent self-rates its output on 5 axes — accuracy, completeness, clarity, actionability, conciseness — with concrete evidence per criterion. Produces a structured 1-5 scorecard with specific improvement suggestions.
+origin: ECC
+---
+
+# Agent Self-Evaluation
+
+After completing a complex task, the agent pauses to rate its own output against a structured 5-axis rubric. This is NOT a pass/fail gate — it's a deliberate reflection step that catches omissions, flags overconfidence, and surface areas for improvement before the user has to.
+
+## When to Activate
+
+- After writing code that spans 3+ files or 50+ lines
+- After completing a multi-step workflow (implement → test → review)
+- After a debugging session that involved 3+ attempts
+- After producing a design document, architecture decision, or written analysis
+- When the user asks "how good was that?" or "rate yourself"
+- At the end of any session Stop hook (if configured — see References)
+
+## Core Concepts
+
+### The 5 Evaluation Axes
+
+| Axis | Question | What it catches |
+|---|---|---|
+| **Accuracy** | Are the facts, claims, and outputs correct? | Hallucinations, wrong API names, incorrect syntax, false statements |
+| **Completeness** | Did it cover everything the user asked for? | Missed edge cases, unhandled error paths, forgotten requirements, skipped subtasks |
+| **Clarity** | Is the explanation understandable and well-structured? | Confusing explanations, jargon without definition, missing context, rambling |
+| **Actionability** | Can the user act on the output immediately? | Vague suggestions, missing steps, "you should X" without showing how, no verification path |
+| **Conciseness** | Did it use the minimum words/tokens needed? | Redundancy, over-explanation, repeating the user's question verbatim, filler content |
+
+### Scoring Scale
+
+```
+5 — Exceptional: no reasonable improvement possible
+4 — Good: minor nits only, no substantive gaps
+3 — Adequate: meets the request but has a notable weakness on at least one axis
+2 — Weak: has a clear gap that affects usability or correctness
+1 — Poor: fundamentally misses the request or contains significant errors
+```
+
+### The Evidence Rule
+
+Every score below 5 MUST cite specific evidence. A score of 3 cannot just say "could be better" — it must say exactly what is missing or wrong. The mantra: **"Show the gap, don't just name it."**
+
+## Workflow
+
+### Step 1: Collect the Raw Material
+
+Gather what you'll evaluate:
+
+```
+- The original user request (read back from conversation)
+- Your final response/output (the deliverable)
+- Any tool outputs that verify correctness (test results, exit codes, lint output)
+- Any user feedback received during the task (corrections, "try again", "that's not right")
+```
+
+### Step 2: Score Each Axis Independently
+
+Work through the 5 axes one at a time. For each:
+
+1. Read the axis question
+2. Find evidence (or lack of evidence) in the output
+3. Assign a score 1-5
+4. If score < 5, write a one-sentence improvement note citing the gap
+
+Do NOT average the scores in your head first and then work backwards. Score each axis fresh.
+
+### Step 3: Produce the Evaluation Report
+
+Use the template from `templates/evaluation-report.md`. The report must include:
+
+```
+- One-line summary
+- 5-axis scorecard (score + evidence per axis)
+- Overall score (simple average, rounded to 1 decimal)
+- 1-3 specific improvements ranked by impact
+- Self-check: "Would the user agree with this assessment?"
+```
+
+### Step 4: Apply the Improvement
+
+If any axis scored 3 or below:
+
+1. State what you would do differently
+2. If the gap is fixable in < 30 seconds (missing link, unclear phrasing), fix it now
+3. If the gap requires rework, flag it explicitly: "This axis scored __ because __. Re-running with __ would likely raise it to __."
+
+## Code Examples
+
+### Example: Good Evaluation (Score 4+)
+
+```
+Task: Add retry logic to HTTP client
+
+Scorecard:
+ Accuracy: 5 — All API calls correct. Verified: retries use
+ exponential backoff. No hallucinated methods.
+ Completeness: 4 — Covered happy path + 3 error cases. Missing:
+ timeout handling for hung connections.
+ Clarity: 5 — Code comments explain backoff formula.
+ PR description links to incident that motivated this.
+ Actionability:5 — Single merge. No follow-up tasks. Tests pass.
+ Conciseness: 4 — 47 lines total. The retry loop could be extracted
+ into a helper to drop ~8 lines.
+
+Overall: 4.6 — One gap (timeout handling). Fix before merging.
+```
+
+### Example: Weak Evaluation (Score 2-3)
+
+```
+Task: Add retry logic to HTTP client
+
+Scorecard:
+ Accuracy: 2 — Used urllib3.Retry which doesn't exist in our
+ httpx-based codebase. Wrong library.
+ Completeness: 3 — Works for GET. POST/PUT not handled (user
+ said "all HTTP requests").
+ Clarity: 4 — Code is readable. Good variable names.
+ Actionability:2 — "Add tests" mentioned but no test file created.
+ User has to write tests before merging.
+ Conciseness: 3 — 120 lines. The retry config is duplicated in
+ 3 places instead of one shared RetryConfig object.
+
+Overall: 2.8 — Wrong library used. Needs httpx rewrite.
+ Fix accuracy first (switch to httpx.Retry), then extend to all
+ HTTP methods, then consolidate config.
+```
+
+## Anti-Patterns
+
+### "Everything is a 5"
+
+```
+❌ Accuracy: 5 — All good.
+ Completeness: 5 — Everything covered.
+ Clarity: 5 — Clear.
+```
+
+No evidence cited. This is self-congratulation, not evaluation. A real 5 requires proving there's nothing to improve.
+
+### Over-penalizing for scope creep
+
+```
+❌ Completeness: 2 — Didn't handle WebSocket connections or
+ gRPC streaming (user didn't ask for these)
+```
+
+Only evaluate against what the user actually requested, not what you could have additionally built.
+
+### Using the evaluation to re-litigate
+
+```
+❌ "As I said earlier, this approach is wrong. Score: 1"
+```
+
+The evaluation is about the delivered output, not about re-arguing design decisions that were already made. If the approach was wrong, that should have been caught before delivery.
+
+### Mixing personal preference with objective gaps
+
+```
+❌ "Score: 3. I don't like Python decorators."
+```
+
+"Don't like" is not evidence. Cite a concrete readability, testability, or correctness concern, or leave the score at 4+.
+
+## Best Practices
+
+- **Evaluate the output, not the process.** The user cares about what you delivered, not how many iterations you took.
+- **One improvement per weak axis.** Don't list 5 things for one axis — pick the highest-impact gap.
+- **Tie improvements to user impact.** "Missing error handling means the user's API call will crash silently" beats "add error handling."
+- **Be specific about what 'fixed' looks like.** "Re-run with httpx.Retry( total=3, backoff_factor=1.0 )" beats "fix the library issue."
+- **Use tool outputs as evidence.** If tests passed, cite them. If lint is clean, cite it. Don't guess — grep for the proof.
+- **If you can't find any gaps, try harder.** A perfect score across all 5 axes is rare. Ask: "If I were the user, what would annoy me about this output?"
+
+## Related Skills
+
+- `agent-eval` — Head-to-head comparison of different coding agents on benchmark tasks
+- `verification-loop` — Systematic verification of outputs against expected results
+- `security-review` — Security-focused code review checklist
diff --git a/skills/agent-self-evaluation/examples/high-score-example.md b/skills/agent-self-evaluation/examples/high-score-example.md
new file mode 100644
index 000000000..a2da55b87
--- /dev/null
+++ b/skills/agent-self-evaluation/examples/high-score-example.md
@@ -0,0 +1,87 @@
+# High-Score Example: Adding Retry Logic
+
+## Task
+> Add retry logic with exponential backoff to all HTTP requests in the API client. Max 3 retries, initial delay 1s, max delay 30s.
+
+## Agent Output
+
+### Summary
+Added `httpx.Retry` transport to the shared `ApiClient` class. All HTTP methods (GET, POST, PUT, DELETE) now retry on 429, 5xx, and connection errors with exponential backoff. 42 tests pass. Single PR ready to merge.
+
+### Files Changed
+- `src/api_client.py` — Added `RetryTransport` wrapper with `httpx.Retry( total=3, backoff_factor=1.0, status_forcelist=[429, 500, 502, 503, 504] )`
+- `tests/test_api_client.py` — Added 8 tests: happy path, retry on 429, retry on 503, max retries exhausted, timeout, backoff timing assertion, idempotent GET retry, no-retry on 4xx (except 429)
+
+### Verification
+```
+$ pytest tests/test_api_client.py -v
+42 passed in 2.34s
+
+$ grep -r "RetryTransport" src/
+src/api_client.py:23: class RetryTransport(httpx.BaseTransport):
+src/api_client.py:45: self._transport = RetryTransport(...)
+
+$ python -c "from src.api_client import ApiClient; c = ApiClient(); print(type(c._transport))"
+<class 'src.api_client.RetryTransport'>
+```
+
+### Edge Cases Handled
+- Connection timeouts (retry with backoff)
+- Rate limiting (429 → retry after delay)
+- Server errors (5xx → retry)
+- Client errors except 429 (no retry — 4xx means the request is bad)
+- Idempotency: GET, PUT, DELETE retry; POST does not (configurable)
+
+### Merge: [PR #423](https://github.com/example/pr/423)
+
+---
+
+## Evaluation Report
+
+```
+============================================================
+AGENT SELF-EVALUATION REPORT
+============================================================
+
+ Accuracy █████ 5/5
+ + Tests passing
+ + Explicit verification (grep confirmed class exists, import test passed)
+ + Lint clean
+ → All claims backed by tool output. No hedging.
+
+ Completeness ████░ 4/5
+ + Edge cases addressed (5 specific scenarios listed)
+ + Error handling present across all HTTP methods
+ → Missing: connection pool exhaustion handling (what happens when all
+ connections are in retry state?) — minor gap, not blocking.
+
+ Clarity █████ 5/5
+ + Uses headings for structure
+ + Uses code blocks
+ + Uses bullet points
+ + Summary in first 3 lines
+ → Well-organized. Reader can scan in 10 seconds.
+
+ Actionability █████ 5/5
+ + PR created and linked
+ + Specific run command given (pytest)
+ + Verification steps included
+ → Single action: merge PR #423. Everything else is done.
+
+ Conciseness ████░ 4/5
+ + No redundancy detected
+ → The verification section could be slightly tighter (3 commands
+ could be 1 with a verification script). Minor.
+
+ OVERALL 4.6/5
+
+TOP IMPROVEMENTS:
+ No axes below 4. Strong output across all dimensions.
+```
+
+### Why This Scores Well
+
+1. **Accuracy pinned to tool output.** Every claim ("tests pass", "class exists", "import works") has a corresponding terminal output line. No "should work" or "probably fine."
+2. **Completeness is explicit about what's covered AND what's not.** The edge cases section lists both handled and intentionally-unhandled cases (POST idempotency).
+3. **Actionability is single-step.** The user only needs to merge one PR. No follow-up tasks, no "then configure X."
+4. **Concision is tight.** The output is ~250 words. The information density is high — every sentence carries weight.
diff --git a/skills/agent-self-evaluation/examples/low-score-example.md b/skills/agent-self-evaluation/examples/low-score-example.md
new file mode 100644
index 000000000..25047e7a8
--- /dev/null
+++ b/skills/agent-self-evaluation/examples/low-score-example.md
@@ -0,0 +1,86 @@
+# Low-Score Example: Adding Retry Logic
+
+## Task
+> Add retry logic with exponential backoff to all HTTP requests in the API client. Max 3 retries, initial delay 1s, max delay 30s.
+
+## Agent Output
+
+Here's a retry implementation. I think this should work for most cases.
+
+I used the urllib3 library since it has built-in retry support. The Retry class handles backoff automatically so we just need to configure it.
+
+```python
+from urllib3.util import Retry
+
+# Add this somewhere in your HTTP client
+retry = Retry(
+ total=3,
+ backoff_factor=1.0,
+ status_forcelist=[500, 502, 503, 504]
+)
+```
+
+You'll want to test this before deploying. There might be edge cases with POST requests but those are probably fine. I haven't tested the timeout behavior but it should work.
+
+Let me know if you need anything else!
+
+---
+
+## Evaluation Report
+
+```
+============================================================
+AGENT SELF-EVALUATION REPORT
+============================================================
+
+ Accuracy ██░░░ 2/5
+ - Hedged claim without verification ("I think this should work")
+ - Untested ("I haven't tested the timeout behavior")
+ - Speculation without evidence ("those are probably fine")
+ → Wrong library used. Project uses httpx, not urllib3.
+ urllib3.util.Retry is incompatible with httpx transport.
+
+ Completeness ███░░ 3/5
+ - Explicit gap acknowledged ("might be edge cases with POST")
+ - Limited scope noted (only mentioned 5xx, not 429 or connection errors)
+ → User asked for "all HTTP requests." Only partial coverage:
+ missing 429 handling, connection errors, timeout handling.
+
+ Clarity ████░ 4/5
+ + Uses code blocks
+ → Code is readable but no explanation of where to add it
+ ("somewhere in your HTTP client" is vague).
+
+ Actionability ██░░░ 2/5
+ - Defers work to user ("you'll want to test this")
+ - Vague suggestion without specifics
+ → No PR, no file created, no test written. User has to:
+ 1. Figure out where to add the code
+ 2. Fix the library mismatch (httpx not urllib3)
+ 3. Write tests
+ 4. Handle POST idempotency
+ 5. Test timeout behavior
+
+ Conciseness ███░░ 3/5
+ - Meta-commentary adds words without information
+ ("Let me know if you need anything else!")
+ → 120 words. Low word count but low information density.
+ Half the text is hedging and disclaimers, not substance.
+
+ OVERALL 2.8/5
+
+TOP IMPROVEMENTS (axes scoring < 4):
+ [Accuracy] Switch to httpx.Retry — grep the codebase to confirm the HTTP
+ library before writing code.
+ [Actionability] Create a PR with the changed file + test file. Run the
+ tests. End with "PR #N ready to merge."
+ [Completeness] List what's covered AND what's not. If POST retry is
+ unsafe, say so explicitly with reasoning.
+```
+
+### Why This Scores Poorly
+
+1. **Accuracy fails at the most basic level** — wrong library. One `grep httpx src/` would have caught this. The hedging language ("I think", "probably", "should work") signals the agent knows it's guessing.
+2. **Not actionable.** The user received a code snippet and a list of things they need to do. The agent did the easy part (suggesting a library) and deferred the hard parts (testing, integration, edge cases) to the user.
+3. **Completeness gaps are acknowledged but not fixed.** "Might be edge cases" is worse than not mentioning them — it shows awareness of the gap and a choice not to address it.
+4. **Information density is low.** 120 words, of which ~60 are hedging/disclaimers/politeness. The actual substance (3 lines of code) could have been delivered in 40 words with verification.
diff --git a/skills/agent-self-evaluation/references/evaluation-criteria.md b/skills/agent-self-evaluation/references/evaluation-criteria.md
new file mode 100644
index 000000000..faf83e7d5
--- /dev/null
+++ b/skills/agent-self-evaluation/references/evaluation-criteria.md
@@ -0,0 +1,71 @@
+# Evaluation Criteria — Detailed Scoring Guide
+
+This reference provides concrete scoring anchors for each axis. Use it when you're unsure whether a gap merits a 4 vs a 3, or a 2 vs a 1.
+
+## Accuracy
+
+| Score | Anchor | Example |
+|---|---|---|
+| 5 | All facts verified against tool output, docs, or authoritative sources. No errors. | Used `httpx.Retry` — confirmed in httpx docs. All method names verified with grep against codebase. |
+| 4 | One minor inaccuracy that doesn't affect correctness. | Correct library, wrong default value for one parameter (httpx defaults to 1.0s, claimed 0.5s). |
+| 3 | One significant factual error, or 3+ minor inaccuracies. | Used `urllib3.Retry` in an httpx codebase. Works in this one case but wrong library. |
+| 2 | Multiple significant errors. Output would fail if followed. | Claimed "add this to package.json" but project uses pyproject.toml. Two other config claims also wrong. |
+| 1 | Fundamentally incorrect. Output contradicts itself or known facts. | Code has syntax errors. API endpoint doesn't exist. Claims a function signature that grep disproves. |
+
+## Completeness
+
+| Score | Anchor | Example |
+|---|---|---|
+| 5 | All explicit and implicit requirements covered. Edge cases handled. Error paths addressed. | User said "add retry to all HTTP requests." GET, POST, PUT, DELETE all covered. Timeout, 429, 5xx all handled. |
+| 4 | All explicit requirements covered. One implicit requirement missed. | All HTTP methods covered. Forgot to handle connection timeouts (not mentioned but expected). |
+| 3 | One explicit requirement missed, or 2+ implicit gaps. | User said "add logging too." Retry logic added but no logging. |
+| 2 | Multiple explicit requirements missed. Output is a partial solution. | Asked for retry + circuit breaker. Only retry implemented. |
+| 1 | Misses the core request. Delivers something adjacent to what was asked. | Asked for retry logic. Wrote a health check endpoint instead. |
+
+## Clarity
+
+| Score | Anchor | Example |
+|---|---|---|
+| 5 | Perfectly structured. Jargon explained or avoided. Visual hierarchy helps scanning. No ambiguity. | README with clear sections, code blocks, and a 10-second summary at top. |
+| 4 | Generally clear. One section could be better organized or one term undefined. | Good structure but `exponential backoff` used without explanation — assumes the reader knows it. |
+| 3 | Understandable after re-reading. Multiple organizational issues or undefined terms. | The explanation circles the point before getting to it. Several terms used before defined. |
+| 2 | Confusing in places. Reader would need to ask follow-up questions. | Code works but the PR description doesn't explain why retry was needed or what it fixes. |
+| 1 | Unintelligible or contradictory. Reader cannot determine what was done or why. | Output is a wall of text with no structure. Conclusions contradict earlier statements. |
+
+## Actionability
+
+| Score | Anchor | Example |
+|---|---|---|
+| 5 | Single action required. Verification path included. No implicit steps. | "Merge this PR. Tests pass: `42 passed`. Deploy with `./deploy.sh`." |
+| 4 | Single action required but verification path is implied, not explicit. | "Merge this PR." (Tests exist but weren't cited. User has to check themselves.) |
+| 3 | Multiple actions required, or one action with unclear next step. | "Review and merge. Then update the config." (Which config? Where? No link or path.) |
+| 2 | User must figure out how to use the output. Missing critical instructions. | Code written but no test file, no run instructions, no PR created. User has to assemble everything. |
+| 1 | Output cannot be acted on without significant rework or clarification. | "Here's a design idea." (No code, no file, no PR. User has to start from scratch.) |
+
+## Conciseness
+
+| Score | Anchor | Example |
+|---|---|---|
+| 5 | Every sentence earns its place. No redundancy. Information density is high. | 30 lines that say what 60 lines would. No repeated points. No filler. |
+| 4 | Minor redundancy. One paragraph could be tightened. | Good overall but repeats the motivation in both the PR description and code comments. |
+| 3 | Noticeable redundancy. 20%+ of content could be removed without loss. | Explains the same concept three times (in summary, body, and conclusion). Verbose examples. |
+| 2 | Significantly bloated. 40%+ of content is filler or repetition. | 200 lines for a task that needed 60. Restates the user's question. Includes irrelevant background. |
+| 1 | Noise-to-signal ratio is inverted. More filler than substance. | 500-line response to a 2-line question. Most of it is boilerplate, repetition, or irrelevant context. |
+
+## Edge Cases
+
+### When the user gave unclear instructions
+
+If the user's request was ambiguous, do NOT penalize completeness for not reading minds. Instead, note in the evaluation: "User's request was ambiguous about __. I chose interpretation __. If they meant __, this score would drop to __."
+
+### When the task is inherently simple
+
+A 3-line bug fix can legitimately score 5/5/5/5/5. The rubric scales with complexity — a simple task done perfectly IS a 5.0. Don't invent gaps to justify lower scores.
+
+### When you caught your own error mid-task
+
+If you made an error, caught it, and fixed it before delivering — that's a 5 on Accuracy for the final output. The evaluation is about what the user received, not your internal process. Note the self-correction as evidence of thoroughness, not as a penalty.
+
+### When the tool output contradicts your claim
+
+If you claimed "tests pass" but the terminal output shows a failure — that's an automatic Accuracy ≤ 2. Tool output is ground truth. Claims without verification are the most common source of low accuracy scores.
diff --git a/skills/agent-self-evaluation/references/hook-integration.md b/skills/agent-self-evaluation/references/hook-integration.md
new file mode 100644
index 000000000..78246b37a
--- /dev/null
+++ b/skills/agent-self-evaluation/references/hook-integration.md
@@ -0,0 +1,59 @@
+# Hook Integration for Session-Stop Self-Evaluation
+
+Add this hook to `hooks/hooks.json` to automatically trigger self-evaluation at the end of every session:
+
+```json
+{
+ "hooks": {
+ "Stop": [
+ {
+ "matcher": "true",
+ "hooks": [
+ {
+ "type": "command",
+ "command": "echo '[Self-Eval] Session complete. Consider running agent-self-evaluation to rate your output.'"
+ }
+ ],
+ "description": "Remind agent to self-evaluate at session end"
+ }
+ ]
+ }
+}
+```
+
+## Integration with the Python Evaluator
+
+The `scripts/evaluate.py` script can be used as a standalone tool:
+
+```bash
+# Pipe agent output directly
+echo "Your agent response here" | python3 skills/agent-self-evaluation/scripts/evaluate.py
+
+# From files
+python3 skills/agent-self-evaluation/scripts/evaluate.py --task task.txt --output response.txt
+```
+
+To integrate it into hooks, capture the last agent output to a file first, then run the evaluator:
+
+```json
+{
+ "PostToolUse": [
+ {
+ "matcher": "tool == \"Bash\" && tool_input.command matches \"(test|pytest|npm test|go test)\"",
+ "hooks": [
+ {
+ "type": "command",
+ "command": "echo '[Self-Eval] Tests completed. Consider running agent-self-evaluation.'"
+ }
+ ],
+ "description": "Remind agent to self-evaluate after test runs"
+ }
+ ]
+}
+```
+
+These hooks are opt-in. Add them to your local `hooks/hooks.json` if you want automated evaluation prompts.
+
+## Manual Usage (Recommended)
+
+The most reliable approach is manual invocation — the agent runs self-evaluation as part of its workflow when the `agent-self-evaluation` skill is active, without requiring hook configuration. The skill's "When to Activate" section already covers trigger conditions (multi-file changes, debugging sessions, design documents).
diff --git a/skills/agent-self-evaluation/scripts/evaluate.py b/skills/agent-self-evaluation/scripts/evaluate.py
new file mode 100755
index 000000000..354f5a7a0
--- /dev/null
+++ b/skills/agent-self-evaluation/scripts/evaluate.py
@@ -0,0 +1,371 @@
+#!/usr/bin/env python3
+"""Standalone agent output evaluator using the 5-axis rubric.
+
+Reads a task description and agent output from stdin or files,
+scores each axis, and prints a structured evaluation report.
+
+Usage:
+ # Pipe output directly
+ echo "Task: Add retry logic" | evaluate.py --output response.txt
+
+ # From files
+ evaluate.py --task task.txt --output response.txt
+
+ # Interactive (reads task from prompt, output from stdin)
+ evaluate.py --interactive
+
+The evaluator uses keyword heuristics + structural checks as a first pass.
+For production use, pair with an LLM judge for semantic understanding.
+"""
+
+import argparse
+import re
+import sys
+from dataclasses import dataclass, field
+from typing import Optional
+
+
+@dataclass
+class AxisScore:
+ name: str
+ score: int
+ evidence: list[str] = field(default_factory=list)
+ improvement: Optional[str] = None
+
+
+def count_words(text: str) -> int:
+ return len(text.split())
+
+
+def check_accuracy(text: str) -> AxisScore:
+ """Check for verifiable claims, tool output references, error signs."""
+ evidence = []
+ deductions = 0
+ score = 5
+
+ # Positive signals: verified claims
+ verified_patterns = [
+ (r"(?i)(tests?\s+pass|all\s+tests?\s+passing|\d+\s+passed)", "Tests passing"),
+ (r"(?i)(exit\s+code\s*[:=]?\s*0|exited\s+with\s+0)", "Clean exit code"),
+ (r"(?i)(lint.*clean|no\s+lint\s+errors|0\s+errors)", "Lint clean"),
+ (r"(?i)(verified|confirmed|validated)\s+(with|against|using|by)", "Explicit verification"),
+ (r"(?i)(grep|rg)\s+.*\b(found|matched|returned)", "Grep confirmed"),
+ ]
+ for pattern, label in verified_patterns:
+ if re.search(pattern, text):
+ evidence.append(f"+ {label}")
+
+ # Negative signals: unverified claims
+ danger_patterns = [
+ (r"(?i)(should\s+work|probably\s+fine|should\s+be\s+ok)", "Hedged claim without verification"),
+ (r"(?i)(I\s+think|I\s+believe|I\s+assume|might\s+be)", "Speculation without evidence"),
+ (r"(?i)(untested|not\s+tested|haven'?t\s+tested)", "Explicitly untested"),
+ (r"(?i)(TODO|FIXME|HACK|WORKAROUND)", "Unresolved TODO/FIXME"),
+ ]
+ for pattern, label in danger_patterns:
+ if re.search(pattern, text):
+ deductions += 1
+ evidence.append(f"- {label}")
+
+ if deductions >= 3:
+ score = 2
+ elif deductions == 2:
+ score = 3
+ elif deductions == 1:
+ score = 4
+
+ if not evidence:
+ evidence.append("No verification signals detected — score assumes correctness")
+
+ result = AxisScore(name="Accuracy", score=score, evidence=evidence)
+ if score < 5:
+ result.improvement = "Cite specific tool outputs (test results, exit codes, grep findings) to back claims"
+ return result
+
+
+def check_completeness(text: str, task: Optional[str] = None) -> AxisScore:
+ """Check for requirement coverage, edge cases, error handling."""
+ evidence = []
+ score = 5
+
+ # Positive signals
+ completeness_signals = [
+ (r"(?i)(edge\s*cases?|corner\s*cases?)", "Edge cases addressed"),
+ (r"(?i)(error\s*handling|exception\s*handling|try/except|try\s*{)", "Error handling present"),
+ (r"(?i)(all\s+\w+\s+(methods|endpoints|routes))", "Full coverage claimed"),
+ (r"(?i)(verification|verified\s+that|confirmed\s+that)", "Verification step present"),
+ ]
+ for pattern, label in completeness_signals:
+ if re.search(pattern, text):
+ evidence.append(f"+ {label}")
+
+ # Gaps
+ gap_signals = [
+ (r"(?i)(not\s+covered|not\s+handled|out\s+of\s+scope)", "Explicit gap acknowledged"),
+ (r"(?i)(only\s+(works|handles|supports)\s+\w+)", "Limited scope noted"),
+ (r"(?i)(assume[sd]?\s+that|assuming\s+the)", "Assumption without verification"),
+ ]
+ deductions = 0
+ for pattern, label in gap_signals:
+ if re.search(pattern, text):
+ deductions += 1
+ evidence.append(f"- {label}")
+
+ if deductions >= 2:
+ score = 3
+ elif deductions == 1:
+ score = 4
+
+ if not evidence:
+ evidence.append("No completeness signals — unable to assess coverage")
+
+ result = AxisScore(name="Completeness", score=score, evidence=evidence)
+ if score < 5:
+ result.improvement = "List what was covered AND what was intentionally excluded, with reasoning"
+ return result
+
+
+def check_clarity(text: str) -> AxisScore:
+ """Check for structure, readability, jargon handling."""
+ evidence = []
+ score = 5
+ deductions = 0
+
+ # Positive signals
+ if re.search(r"^#{1,3}\s+", text, re.MULTILINE):
+ evidence.append("+ Uses headings for structure")
+ if re.search(r"```", text):
+ evidence.append("+ Uses code blocks")
+ if re.search(r"^\s*[-*]\s+", text, re.MULTILINE):
+ evidence.append("+ Uses bullet points")
+
+ # Negative signals
+ # Wall of text: long paragraph without breaks
+ paragraphs = [p for p in text.split("\n\n") if p.strip()]
+ for p in paragraphs:
+ if count_words(p) > 200:
+ deductions += 1
+ evidence.append("- Wall-of-text paragraph (>200 words without break)")
+ break
+
+ # Jargon without definition
+ jargon = [
+ (r"\b(idempotent|race condition|deadlock|thundering herd)\b", "concurrency"),
+ (r"\b(exponential backoff|circuit breaker|bulkhead)\b", "resilience"),
+ (r"\b(ACID|CAP|eventual consistency|linearizability)\b", "database theory"),
+ ]
+ for pattern, domain in jargon:
+ if re.search(pattern, text, re.IGNORECASE):
+ if not re.search(rf"(?i)({domain}|means|refers to|i\.e\.|in other words)", text):
+ deductions += 1
+ evidence.append(f"- Domain term used without explanation ({domain})")
+ break
+
+ if not any(t in text[:100].lower() for t in ["summary", "tldr", "overview", "in short"]):
+ # No early summary — penalize only if text is long
+ if count_words(text) > 300:
+ deductions += 1
+ evidence.append("- No summary/TLDR in first 100 words (text is 300+ words)")
+
+ if deductions >= 3:
+ score = 2
+ elif deductions == 2:
+ score = 3
+ elif deductions == 1:
+ score = 4
+
+ if not evidence:
+ evidence.append("+ Well-structured with no clarity issues detected")
+
+ result = AxisScore(name="Clarity", score=score, evidence=evidence)
+ if score < 5:
+ result.improvement = "Add headings, break long paragraphs, define domain terms on first use"
+ return result
+
+
+def check_actionability(text: str) -> AxisScore:
+ """Check if the user can act on the output immediately."""
+ evidence = []
+ score = 5
+ deductions = 0
+
+ # Positive signals
+ actionable_signals = [
+ (r"(?i)(merge|PR|pull request).*?(created|ready|open)", "PR created"),
+ (r"(?i)(run|execute)\s+[`\"']?[\w./-]+", "Specific run command given"),
+ (r"(?i)(next\s+steps?|follow[- ]up|what\s+to\s+do)", "Next steps provided"),
+ (r"(?i)(file\s+(created|written|modified|updated)\s+at)", "File path specified"),
+ ]
+ for pattern, label in actionable_signals:
+ if re.search(pattern, text):
+ evidence.append(f"+ {label}")
+
+ # Negative signals
+ vague_signals = [
+ (r"(?i)(you\s+(should|could|might\s+want\s+to))\s+\w+", "Vague suggestion without specifics"),
+ (r"(?i)(consider|maybe|perhaps)\s+\w+ing", "Non-committal suggestion"),
+ (r"(?i)(figure\s+out|look\s+into|investigate)\s", "Defers work to user"),
+ ]
+ for pattern, label in vague_signals:
+ if re.search(pattern, text):
+ deductions += 1
+ evidence.append(f"- {label}")
+
+ if deductions >= 3:
+ score = 2
+ elif deductions == 2:
+ score = 3
+ elif deductions == 1:
+ score = 4
+
+ if not evidence:
+ evidence.append("No actionability signals — user may need to ask 'what now?'")
+
+ result = AxisScore(name="Actionability", score=score, evidence=evidence)
+ if score < 5:
+ result.improvement = "End with a single clear action: 'Merge this PR', 'Run ./deploy.sh', or 'Review the 3 changed files'"
+ return result
+
+
+def check_concision(text: str, task: Optional[str] = None) -> AxisScore:
+ """Check for redundancy, filler, information density."""
+ evidence = []
+ score = 5
+ wc = count_words(text)
+
+ # Heuristic: task-to-output ratio
+ if task:
+ task_wc = count_words(task)
+ ratio = wc / max(task_wc, 1)
+ if ratio > 15:
+ evidence.append(f"- Output is {ratio:.0f}x longer than task description (high ratio)")
+ score = min(score, 3)
+ elif ratio > 8:
+ evidence.append(f"- Output is {ratio:.0f}x longer than task description")
+ score = min(score, 4)
+
+ # Redundancy signals
+ redundancy_checks = [
+ (r"(?i)(as\s+(I|we)\s+(mentioned|said|noted|discussed)\s+(earlier|above|before))",
+ "Refers back to earlier statement (possible repetition)"),
+ (r"(?i)(to\s+summarize|in\s+summary|in\s+conclusion|to\s+conclude)",
+ "Has explicit summary (good if needed, flag if redundant)"),
+ (r"(?i)(let\s+me\s+(explain|break\s+this\s+down|walk\s+you\s+through))",
+ "Meta-commentary adds words without information"),
+ ]
+ redundant_count = 0
+ for pattern, label in redundancy_checks:
+ matches = re.findall(pattern, text)
+ if len(matches) > 2:
+ redundant_count += 1
+ evidence.append(f"- '{label}' appears {len(matches)} times")
+
+ if redundant_count >= 2:
+ score = min(score, 3)
+ elif redundant_count == 1:
+ score = min(score, 4)
+
+ if not evidence and score == 5:
+ evidence.append("+ No redundancy detected. Information density appears good.")
+
+ result = AxisScore(name="Conciseness", score=score, evidence=evidence)
+ if score < 5:
+ result.improvement = "Cut meta-commentary, remove repeated points, trim examples to one representative case"
+ return result
+
+
+def evaluate(task: Optional[str], output: str) -> list[AxisScore]:
+ """Run all 5 axis checks and return scored results."""
+ return [
+ check_accuracy(output),
+ check_completeness(output, task),
+ check_clarity(output),
+ check_actionability(output),
+ check_concision(output, task),
+ ]
+
+
+def format_report(scores: list[AxisScore]) -> str:
+ """Format scores into a readable evaluation report."""
+ avg = sum(s.score for s in scores) / len(scores)
+ lines = []
+ lines.append("=" * 60)
+ lines.append("AGENT SELF-EVALUATION REPORT")
+ lines.append("=" * 60)
+ lines.append("")
+
+ for s in scores:
+ bar = "█" * s.score + "░" * (5 - s.score)
+ lines.append(f" {s.name:<15} {bar} {s.score}/5")
+ for e in s.evidence:
+ lines.append(f" {e}")
+ if s.improvement:
+ lines.append(f" → {s.improvement}")
+ lines.append("")
+
+ lines.append(f" {'OVERALL':<15} {avg:.1f}/5")
+ lines.append("")
+
+ # Top improvements
+ improvements = [(s, s.improvement) for s in scores if s.improvement and s.score < 4]
+ if improvements:
+ lines.append("TOP IMPROVEMENTS (axes scoring < 4):")
+ for s, imp in sorted(improvements, key=lambda x: x[0].score):
+ lines.append(f" [{s.name}] {imp}")
+ else:
+ lines.append("No axes below 4. Strong output across all dimensions.")
+
+ return "\n".join(lines)
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Evaluate agent output against the 5-axis rubric"
+ )
+ parser.add_argument("--task", help="Task description (file path or inline text)")
+ parser.add_argument("--output", help="Agent output to evaluate (file path)")
+ parser.add_argument("--interactive", action="store_true", help="Prompt for task and read output from stdin")
+ args = parser.parse_args()
+
+ task = None
+ output = None
+
+ if args.interactive:
+ task = input("Task description: ").strip()
+ print("Paste agent output (Ctrl+D to finish):")
+ output = sys.stdin.read()
+ elif args.task and args.output:
+ # Read task
+ try:
+ with open(args.task) as f:
+ task = f.read()
+ except FileNotFoundError:
+ task = args.task # Treat as inline text
+
+ # Read output
+ try:
+ with open(args.output) as f:
+ output = f.read()
+ except FileNotFoundError:
+ print(f"Error: output file '{args.output}' not found", file=sys.stderr)
+ sys.exit(1)
+ else:
+ # Pipe mode: read output from stdin
+ output = sys.stdin.read()
+ if args.task:
+ try:
+ with open(args.task) as f:
+ task = f.read()
+ except FileNotFoundError:
+ task = args.task
+
+ if not output:
+ print("Error: no output to evaluate", file=sys.stderr)
+ sys.exit(1)
+
+ scores = evaluate(task, output)
+ print(format_report(scores))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/skills/agent-self-evaluation/templates/evaluation-report.md b/skills/agent-self-evaluation/templates/evaluation-report.md
new file mode 100644
index 000000000..ce29f1ce7
--- /dev/null
+++ b/skills/agent-self-evaluation/templates/evaluation-report.md
@@ -0,0 +1,70 @@
+# Agent Self-Evaluation Report Template
+
+Copy this template and fill in after completing a task. Keep it in your conversation — the user sees it inline.
+
+```
+============================================================
+AGENT SELF-EVALUATION REPORT
+============================================================
+
+ Accuracy █████ 5/5 or ███░░ 3/5
+ + [Evidence: passing tests, verified claims]
+ - [Gaps: unverified claims, hedging language]
+
+ Completeness █████ 5/5
+ + [What's covered: all requirements + edge cases]
+ - [What's missing: explicitly acknowledge gaps]
+
+ Clarity █████ 5/5
+ + [Structure: headings, code blocks, bullet points]
+ - [Issues: undefined terms, wall of text, no summary]
+
+ Actionability █████ 5/5
+ + [User can: merge PR, run command, review file]
+ - [Blockers: missing steps, vague suggestions]
+
+ Conciseness █████ 5/5
+ + [Tight: no repetition, high information density]
+ - [Bloat: filler, meta-commentary, repeated points]
+
+ OVERALL X.X/5
+
+TOP IMPROVEMENTS:
+ [Only list axes scoring < 4, ranked by user impact]
+```
+
+## Quick Reference: Scoring Triggers
+
+| If you see this... | Accuracy | Completeness | Clarity | Actionability | Conciseness |
+|---|---|---|---|---|---|
+| "should work" / "probably fine" | ≤4 | — | — | — | — |
+| "I think" / "I believe" | ≤4 | — | — | — | — |
+| No test output cited | ≤4 | — | — | — | — |
+| "TODO" / "FIXME" left behind | ≤3 | ≤3 | — | ≤3 | — |
+| Missing error handling | — | ≤3 | — | — | — |
+| Only happy path covered | — | ≤3 | — | — | — |
+| Wall-of-text paragraph (>200 words) | — | — | ≤3 | — | — |
+| No headings or structure | — | — | ≤3 | — | — |
+| "You should..." without specifics | — | — | — | ≤3 | — |
+| No PR or file created | — | — | — | ≤3 | — |
+| User needs to figure out next step | — | — | — | ≤2 | — |
+| Repeated points (3+ times) | — | — | — | — | ≤3 |
+| "Let me explain..." / "To summarize..." x3+ | — | — | — | — | ≤3 |
+| Output >15x longer than task | — | — | — | — | ≤3 |
+
+## When to Skip
+
+Skip the evaluation if:
+- Task was a single tool call (e.g., "read this file" — nothing to evaluate)
+- User explicitly says "don't evaluate" or "just do it"
+- Task is purely conversational (greeting, small talk)
+- You're mid-workflow and the user will judge the final output, not intermediate steps
+
+## Post-Evaluation Actions
+
+| Overall Score | What to do |
+|---|---|
+| ≥4.5 | Deliver. No changes needed. |
+| 3.5–4.4 | Flag the top improvement but deliver. Fix if <30 seconds. |
+| 2.5–3.4 | State what you'd change. Ask user: "Should I redo [axis] or deliver as-is?" |
+| <2.5 | Don't deliver. Say: "This scored __ because __. Let me redo this with [specific fix]." Then redo. |