The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
feat(skills): add evalview-agent-testing skill and MCP server (#828)
Hidai Bar-Mor0f40fd030c22
diff --git a/mcp-configs/mcp-servers.json b/mcp-configs/mcp-servers.jsonindex 8c0b02b8a..c62d88668 100644--- a/mcp-configs/mcp-servers.json+++ b/mcp-configs/mcp-servers.json@@ -152,6 +152,14 @@"CONFLUENCE_API_TOKEN": "YOUR_CONFLUENCE_TOKEN_HERE"},"description": "Confluence Cloud integration — search pages, retrieve content, explore spaces"+ },+ "evalview": {+ "command": "python3",+ "args": ["-m", "evalview", "mcp", "serve"],+ "env": {+ "OPENAI_API_KEY": "YOUR_OPENAI_API_KEY_HERE"+ },+ "description": "AI agent regression testing — snapshot behavior, detect regressions in tool calls and output quality. 8 tools: create_test, run_snapshot, run_check, list_tests, validate_skill, generate_skill_tests, run_skill_test, generate_visual_report. API key optional — deterministic checks (tool diff, output hash) work without it. Install: pip install \"evalview>=0.5,<1\""}},"_comments": {diff --git a/skills/evalview-agent-testing/SKILL.md b/skills/evalview-agent-testing/SKILL.mdnew file mode 100644index 000000000..326d7a1a0--- /dev/null+++ b/skills/evalview-agent-testing/SKILL.md@@ -0,0 +1,160 @@+---+name: evalview-agent-testing+description: Regression testing for AI agents using EvalView. Snapshot agent behavior, detect regressions in tool calls and output quality, and block broken agents before production.+origin: ECC+tools: Bash, Read, Write+---++# EvalView Agent Testing++Automated regression testing for AI agents. EvalView snapshots your agent's behavior (tool calls, parameters, sequence, output), then diffs against the baseline after every change. When something breaks, you know immediately — before it ships.++## When to Activate++- After modifying agent code, prompts, or tool definitions+- After a model update or provider change+- Before deploying an agent to production+- When setting up CI/CD for an agent project+- When an autonomous loop (OpenClaw, coding agents) needs a fitness function+- When agent output changes unexpectedly and you need to identify what shifted++## Core Workflow++```bash+# 1. Set up+pip install "evalview>=0.5,<1"+evalview init # Detect agent, create starter test suite++# 2. Baseline+evalview snapshot # Save current behavior as golden baseline++# 3. Gate every change+evalview check # Diff against baseline — catches regressions++# 4. Monitor in production+evalview monitor --slack-webhook https://hooks.slack.com/services/...+```++## Understanding Check Results++| Status | Meaning | Action |+|--------|---------|--------|+| `PASSED` | Behavior matches baseline | Ship with confidence |+| `TOOLS_CHANGED` | Different tools called | Review the diff |+| `OUTPUT_CHANGED` | Same tools, output shifted | Review the diff |+| `REGRESSION` | Score dropped significantly | Fix before shipping |++## Python API for Autonomous Loops++Use `gate()` as a programmatic regression gate inside agent frameworks, autonomous coding loops, or CI scripts:++```python+from evalview import gate, DiffStatus++# Full evaluation+result = gate(test_dir="tests/")+if not result.passed:+ for d in result.diffs:+ if not d.passed:+ delta = f" ({d.score_delta:+.1f})" if d.score_delta is not None else ""+ print(f" {d.test_name}: {d.status.value}{delta}")++# Quick mode — no LLM judge, $0, sub-second+result = gate(test_dir="tests/", quick=True)+```++### Auto-Revert on Regression++```python+from evalview.openclaw import gate_or_revert++# In an autonomous coding loop:+make_code_change()+if not gate_or_revert("tests/", quick=True):+ # Change was automatically reverted+ try_alternative_approach()+```++> **Warning:** `gate_or_revert` runs `git checkout -- .` when a regression is detected, discarding uncommitted changes. Commit or stash work-in-progress before entering the loop. You can also pass a custom revert command: `gate_or_revert("tests/", revert_cmd="git stash")`.++## MCP Integration++EvalView exposes 8 tools via MCP — works with Claude Code, Cursor, and any MCP client:++```bash+claude mcp add --transport stdio evalview -- evalview mcp serve+```++Tools: `create_test`, `run_snapshot`, `run_check`, `list_tests`, `validate_skill`, `generate_skill_tests`, `run_skill_test`, `generate_visual_report`++After connecting, Claude Code can proactively check for regressions after code changes:+- "Did my refactor break anything?" triggers `run_check`+- "Save this as the new baseline" triggers `run_snapshot`+- "Add a test for the weather tool" triggers `create_test`++## CI/CD Integration++```yaml+# .github/workflows/evalview.yml+name: Agent Regression Check+on: [pull_request, push]+jobs:+ check:+ runs-on: ubuntu-latest+ env:+ OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}+ steps:+ - uses: actions/checkout@v4+ - run: pip install "evalview>=0.5,<1"+ - run: evalview check --fail-on REGRESSION+```++`--fail-on REGRESSION` gates on score drops only. For stricter gating that also catches tool sequence changes, use `--fail-on REGRESSION,TOOLS_CHANGED` or `--strict` (fails on any change).++## Test Case Format++```yaml+name: refund-flow+input:+ query: "I need a refund for order #4812"+expected:+ tools: ["lookup_order", "check_refund_policy", "issue_refund"]+ forbidden_tools: ["delete_order"]+ output:+ contains: ["refund", "processed"]+ not_contains: ["error"]+thresholds:+ min_score: 70+```++Multi-turn tests are also supported:++```yaml+name: clarification-flow+turns:+ - query: "I want a refund"+ expected:+ output:+ contains: ["order number"]+ - query: "Order 4812"+ expected:+ tools: ["lookup_order", "issue_refund"]+```++## Best Practices++- **Snapshot after every intentional change.** Baselines should reflect intended behavior.+- **Use `--preview` before snapshotting.** `evalview snapshot --preview` shows what would change without saving.+- **Quick mode for tight loops.** `gate(quick=True)` skips the LLM judge — free and fast for iterative development.+- **Full evaluation for final validation.** Run without `quick=True` before deploying to get LLM-as-judge scoring.+- **Commit `.evalview/golden/` to git.** Baselines should be versioned. Don't commit `state.json`.+- **Use variants for non-deterministic agents.** `evalview snapshot --variant v2` stores alternate valid behaviors (up to 5).+- **Monitor in production.** `evalview monitor` catches gradual drift that individual checks miss.++## Installation++```bash+pip install "evalview>=0.5,<1"+```++Package: [evalview on PyPI](https://pypi.org/project/evalview/)