Harness engineering beginner tutorial, from 0 to 1
TypeScript agentagenticagentic-aiaiai-agentai-agentsdshdsh-pluginharnessharness-engineeringharness-frameworkllm
feat: Add harness-creator skill for harness creation
StLeoX6b59d87f2db4
diff --git a/.opencode/skills/harness-creator/README.md b/.opencode/skills/harness-creator/README.mdnew file mode 100644index 0000000..0133cdd--- /dev/null+++ b/.opencode/skills/harness-creator/README.md@@ -0,0 +1,193 @@+# harness-creator Skill++Production harness engineering skill for AI coding agents, distilled from Learn Harness Engineering course and industry best practices.++## Installation++```bash+npx skills add github:harness-creator+```++Or manually copy the `.opencode/skills/harness-creator/` directory to your skill path.++## What This Skill Does++This skill helps you:+- **Create harnesses from scratch** — AGENTS.md, feature lists, verification workflows+- **Improve existing harnesses** — Five-subsystem assessment with prioritized improvements+- **Design session continuity** — Memory persistence, progress tracking, handoff procedures+- **Benchmark effectiveness** — Before/after comparison with quantitative metrics+- **Apply production patterns** — Memory, context engineering, tool safety, multi-agent coordination++## Core Framework: Five Subsystems++Every harness consists of five subsystems:++1. **Instructions** — AGENTS.md as routing layer, progressive disclosure via docs/ hierarchy+2. **State** — feature_list.json, progress.md, session handoff files+3. **Verification** — Explicit commands that agent MUST run before claiming done+4. **Scope** — One-feature-at-a-time policy, clear definition of done+5. **Lifecycle** — init.sh, clean-state checklists, session continuity mechanisms++## Reference Patterns++This skill includes 6 deep-dive reference documents:++| Pattern | When to Use |+|---------|-------------|+| [Memory Persistence](references/memory-persistence-pattern.md) | Agent forgets between sessions, need persistent project knowledge |+| [Context Engineering](references/context-engineering-pattern.md) | Context budget management, JIT loading, delegation isolation |+| [Tool Registry](references/tool-registry-pattern.md) | Tool safety, concurrency control, permission pipelines |+| [Multi-Agent Coordination](references/multi-agent-pattern.md) | Parallelism, specialization, researcher→implementer workflows |+| [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) | Hooks, background tasks, initialization sequences |+| [Gotchas](references/gotchas.md) | 15 non-obvious failure modes with fixes |++## Usage Examples++### Create Minimal Harness++```+User: "I need to set up AGENTS.md for my TypeScript project"++Skill will:+1. Ask about project context (stack, size, agent tool)+2. Generate AGENTS.md with startup workflow and working rules+3. Create feature_list.json template with placeholder features+4. Create init.sh with verification commands+5. Explain how to use each file+```++### Assess Existing Harness++```+User: "My agent still breaks things even with AGENTS.md"++Skill will:+1. Request current AGENTS.md content+2. Score each of 5 subsystems (1-5 scale)+3. Identify lowest-scoring subsystem as bottleneck+4. Provide prioritized improvement plan with concrete steps+```++### Design Session Continuity++```+User: "Agent forgets everything between sessions"++Skill will:+1. Explain memory layers (instruction vs auto-memory)+2. Design progress.md template for session tracking+3. Create session-handoff.md structure+4. Implement two-step save invariant (topic file → index)+```++## When to Trigger++This skill triggers on:+- "Create AGENTS.md / CLAUDE.md"+- "Improve agent reliability"+- "Agent forgets between sessions"+- "Multi-session continuity needed"+- "Benchmark harness effectiveness"+- "Design verification workflow"+- "Memory persistence patterns"+- "Context engineering for agents"++## When NOT to Use++This skill does NOT cover:+- Prompt engineering or system prompt design+- Model selection or fine-tuning+- Generic software architecture+- LLM API integration basics++## Templates Included++- `templates/agents.md` — AGENTS.md scaffold with working rules+- `templates/feature-list.json` — JSON Schema + example+- `templates/init.sh` — Standard initialization script+- `templates/progress.md` — Session progress log template+- `templates/session-handoff.md` — Handoff structure++## EvaluationFramework++5 test cases in `evals/evals.json`:+1. **Minimal Harness Creation** — Full setup from scratch+2. **Session Continuity Setup** — Memory and handoff design+3. **Harness Assessment** — Five-subsystem scoring+4. **Verification Workflow Design** — Force agent to verify before done+5. **Memory Taxonomy Design** — What to save vs skip++Run evaluation with skill-creator framework for quantitative benchmarks.++## Compatibility++- **Agents**: Claude Code, Codex, Cursor, Windsurf, generic+- **License**: MIT+- **Languages**: English / 中文 (bilingual support in SKILL.md)++## 兼容性++- **代理工具**: Claude Code, Codex, Cursor, Windsurf, generic+- **许可证**: MIT+- **语言**: 英文 / 中文 (SKILL.md 中双语支持)++## Project Structure++```+harness-creator/+├── SKILL.md # Main skill definition+├── metadata.json # Skill metadata, triggers, compatibility+├── evals/+│ └── evals.json # 5 test cases with expectations+├── templates/+│ ├── agents.md # AGENTS.md template+│ ├── feature-list.json # Feature tracker template+│ ├── init.sh # Initialization script+│ └── progress.md # Session progress template+└── references/+ ├── memory-persistence-pattern.md+ ├── context-engineering-pattern.md+ ├── tool-registry-pattern.md+ ├── multi-agent-pattern.md+ ├── lifecycle-bootstrap-pattern.md+ └── gotchas.md # 15 failure modes+```++## Development Roadmap++- [x] Chinese translation in SKILL.md (bilingual support added)+- [ ] Python scripts for automated harness generation+- [ ] HTML viewer for harness assessment results+- [ ] Expanded eval set (10+ test cases)+- [ ] Integration with skill-creator benchmark framework+- [ ] Full Chinese localization (harness-creator-zh directory)++## 开发路线图++- [x] SKILL.md 中文翻译(已添加双语支持)+- [ ] Harness 自动生成 Python 脚本+- [ ] Harness 评估结果 HTML 查看器+- [ ] 扩展测试用例(10+ 个)+- [ ] 集成 skill-creator 基准测试框架+- [ ] 完整中文本地化(harness-creator-zh 目录)++## Contributing++Issues and PRs welcome. Key areas for contribution:+- Additional reference patterns (skill runtime, hook lifecycle, etc.)+- More eval test cases covering edge cases+- Script automation for common harness tasks+- Case studies from production deployments++## License++MIT — See LICENSE file for details.++## Acknowledgments++This skill synthesizes:+- Learn Harness Engineering course framework+- OpenAI Harness Engineering principles+- Anthropic effective harnesses research+- Agentic Harness Patterns skill (pattern extraction methodology)diff --git a/.opencode/skills/harness-creator/SKILL.md b/.opencode/skills/harness-creator/SKILL.mdnew file mode 100644index 0000000..3149199--- /dev/null+++ b/.opencode/skills/harness-creator/SKILL.md@@ -0,0 +1,440 @@+---+name: harness-creator+description: >-+ Harness engineering for AI coding agents — five subsystems, memory persistence,+ session continuity, verification workflows, scope control, lifecycle management.+when_to_use: >-+ Use whenever: building harness from scratch, improving agent reliability, agent forgets+ between sessions, agent overreach or scope creep, broken tests after agent work,+ multi-session continuity needed, verification gaps, audit harness quality, benchmark+ effectiveness, create AGENTS.md/CLAUDE.md, design feature tracking, session handoff.+license: MIT+---++# Harness Creator++Production harness engineering for AI coding agents.++**For:** Engineers building or extending coding-agent runtimes, custom agents, multi-session workflows, or anyone who wants their agent to work reliably across sessions.++**Not for:** Prompt engineering, model selection, generic software architecture, or one-off agent tasks.++All principles are grounded in the Learn Harness Engineering framework and production agent runtime decisions.++---++# Harness Creator(中文版)++面向 AI 编程代理的生产级 Harness 工程技能。++**适用人群:** 构建或扩展编程代理运行时、自定义代理、多会话工作流的工程师,或任何希望代理跨会话可靠工作的人。++**不适用场景:** 提示工程、模型选择、通用软件架构或一次性代理任务。++所有原则均基于 Learn Harness Engineering 框架和生产代理运行时决策。++---++## Choose Your Problem++| If you want to... | Read |+|---|---|+| Make the agent remember corrections and project rules between sessions | [Memory Persistence](references/memory-persistence-pattern.md) |+| Package reusable workflows and domain knowledge | [Skill Runtime](references/skill-runtime-pattern.md) |+| Let the agent work powerfully but not dangerously | [Tool Registry & Safety](references/tool-registry-pattern.md) |+| Give the agent the right context at the right cost | [Context Engineering](references/context-engineering-pattern.md) |+| Split work across multiple agents without chaos | [Multi-agent Coordination](references/multi-agent-pattern.md) |+| Extend behavior with hooks, background tasks, startup logic | [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) |+| Build the complete 5-subsystem harness | [Five Subsystems Guide](#the-five-subsystem-harness-framework) |++**Before you start building:** Read the [Gotchas](#gotchas) — these are the non-obvious failure modes that cost the most time.++---++## 选择你要解决的问题++| 如果你想... | 阅读 |+|---|---|+| 让代理在会话之间记住修正和项目规则 | [记忆持久化](references/memory-persistence-pattern.md) |+| 打包可重复使用的工作流和领域知识 | [技能运行时](references/skill-runtime-pattern.md) |+| 让代理强大但安全地工作 | [工具注册与安全](references/tool-registry-pattern.md) |+| 以合适的成本给代理合适的上下文 | [上下文工程](references/context-engineering-pattern.md) |+| 在多个代理之间分配工作而不混乱 | [多代理协调](references/multi-agent-pattern.md) |+| 使用 hooks、后台任务、启动逻辑扩展行为 | [生命周期与引导](references/lifecycle-bootstrap-pattern.md) |+| 构建完整的 5 子系统 harness | [五子系统指南](#the-five-subsystem-harness-framework) |++**开始构建之前:** 阅读 [陷阱](references/gotchas.md) — 这些是最耗时的非明显失败模式。++---++## The Five-Subsystem Harness Framework++**Every harness consists of five subsystems:**++1. **Instructions (Recipe Shelf)**: AGENTS.md, CLAUDE.md, docs/ hierarchy+2. **State (Prep Station)**: feature_list.json, progress.md, session-handoff.md+3. **Verification (Quality Check Window)**: Verification commands, test suites, type checks+4. **Scope (Task Boundaries)**: One-feature-at-a-time policies, definition of done+5. **Lifecycle (Session Management)**: init.sh, clean-state checklists, handoff procedures++**When creating or improving a harness, systematically address each subsystem.**++---++## Creating a Harness++### Phase 1: Context Gathering++Start by understanding the user's situation:++1. **What project is this for?** (tech stack, size, complexity)+2. **What agent tool are they using?** (Claude Code, Codex, Cursor, etc.)+3. **What exists already?** (any AGENTS.md, progress tracking, verification?)+4. **What problems are they experiencing?** (agent overreach, lost context, broken tests?)+5. **What's the team's tolerance for structure?** (minimal vs. comprehensive)++If the user hasn't provided this context, ask before proceeding.++### Phase 2: Harness Assessment (Existing Projects)++If the user has an existing harness, assess it using the five-tuple framework:++For each subsystem, score 1-5:+- **5**: Exemplary, documented, consistently followed+- **4**: Good, mostly complete, occasional gaps+- **3**: Adequate, covers basics, missing polish+- **2**: Weak, incomplete, inconsistently applied+- **1**: Missing or actively harmful++Identify the lowest-scoring subsystem — that's the bottleneck. Focus improvement efforts there first.++### Phase 3: Design++Based on the assessment, design the harness components:++**Instructions:**+- Create a short AGENTS.md (~50-100 lines) as the routing layer+- Link to detailed docs in docs/ directory (ARCHITECTURE.md, PRODUCT.md, etc.)+- Define startup workflow: what the agent reads before coding++**State:**+- Create feature_list.json with feature definitions and status tracking+- Create or update progress.md for session continuity+- Design session-handoff.md template if needed++**Verification:**+- List explicit verification commands in AGENTS.md+- Ensure init.sh runs verification+- Design quality score tracking if appropriate++**Scope:**+- Define one-feature-at-a-time policy+- Document feature dependencies+- Create definition of done checklist++**Lifecycle:**+- Create init.sh for initialization+- Design clean-state checklist+- Document session handoff procedure++### Phase 4: Implementation++Create the harness files. Use bundled scripts where available:++```bash+# Use bundled scripts from scripts/ directory+# (See scripts/ section for available tools)+```++### Phase 5: Testing and Benchmarking++Test the harness with real agent sessions:++1. **Baseline**: Run a representative task without the harness+2. **With Harness**: Run the same task with the harness+3. **Measure**: Success rate, time, token usage, rework+4. **Compare**: Quantify the improvement++For rigorous benchmarking, see the "Running Benchmarks" section below.++---++## Harness File Templates++### AGENTS.md Structure++A minimal AGENTS.md should include:++```markdown+# AGENTS.md++[One-sentence project purpose]++## Startup Workflow++Before writing code:+1. [Step 1: e.g., Read this file]+2. [Step 2: e.g., Read ARCHITECTURE.md]+3. [Step 3: e.g., Run ./init.sh]+4. [Step 4: e.g., Read feature_list.json]++## Working Rules++- [Rule 1: e.g., One feature at a time]+- [Rule 2: e.g., Verification required before claiming done]+- [Rule 3: e.g., Update progress before ending session]++## Required Artifacts++- `feature_list.json`: Feature state tracker+- `progress.md`: Session continuity log+- `init.sh`: Standard startup and verification++## Definition of Done++A feature is done when:+- [ ] Implementation complete+- [ ] Verification passed+- [ ] Evidence recorded+- [ ] Repository restartable++## End of Session++Before ending:+1. Update progress.md+2. Update feature_list.json+3. Record blockers/risks+4. Commit with descriptive message+5. Leave clean restart path+```++### feature_list.json Structure++```json+{+ "features": [+ {+ "id": "feat-001",+ "name": "Document Import",+ "description": "Allow users to import PDF and TXT documents",+ "dependencies": [],+ "status": "done",+ "evidence": "tests pass, manual verification on 2024-01-15"+ },+ {+ "id": "feat-002",+ "name": "Document Chunking",+ "description": "Split documents into ~500 char chunks with metadata",+ "dependencies": ["feat-001"],+ "status": "in-progress",+ "evidence": ""+ }+ ]+}+```++### init.sh Structure++```bash+#!/bin/bash+set -e++echo "=== Installing dependencies ==="+npm install++echo "=== Running type check ==="+npm run check++echo "=== Running tests ==="+npm test++echo "=== Building application ==="+npm run build++echo "=== Verification complete ==="+```++---++## Running Benchmarks++To measure harness effectiveness:++### Step 1: Define Representative Tasks++Pick 2-3 tasks that are:+- Real work the user would actually do+- Challenging enough to fail without proper harness+- Verifiable (clear success criteria)++### Step 2: Run Comparative Sessions++For each task:+- **Without Harness**: Run the task on a clean repo copy+- **With Harness**: Run the same task with the harness in place++Record:+- Success/failure+- Time taken+- Token usage+- Rework required+- Session restarts needed++### Step 3: Aggregate Results++Calculate:+- Success rate improvement+- Time efficiency change+- Token efficiency change+- Qualitative feedback++### Step 4: Iterate++Use results to identify:+- Which harness components add most value+- Which components are over-engineered+- Where to focus improvement efforts++---++## Bundled Resources++### References (Deep-Dive Patterns)++| Document | Covers |+|----------|--------|+| [Memory Persistence](references/memory-persistence-pattern.md) | Four-level instruction hierarchy, auto-memory taxonomy, background extraction |+| [Context Engineering](references/context-engineering-pattern.md) | Select / Compress / Isolate / Write operations, budget management |+| [Tool Registry](references/tool-registry-pattern.md) | Fail-closed registration, per-call concurrency, permission pipeline |+| [Multi-Agent](references/multi-agent-pattern.md) | Coordinator / Fork / Swarm patterns, context sharing |+| [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) | Hook system, long-running tasks, dependency-ordered init |+| [Gotchas](references/gotchas.md) | 15 non-obvious failure modes with fixes |++### Templates++- `templates/agents.md` — AGENTS.md / CLAUDE.md skeleton+- `templates/feature-list.json` — Feature state tracker+- `templates/init.sh` — Standard initialization script+- `templates/progress.md` — Session progress log+- `templates/session-handoff.md` — Session handoff template++### Scripts (Optional)++- `scripts/create-harness.ts` — Generate harness files from templates+- `scripts/validate-harness.ts` — Check harness completeness+- `scripts/run-benchmark.ts` — Execute harness effectiveness comparison++---++## Gotchas++Non-obvious principles that will cause bugs if you violate them:++1. **Memory index caps fire silently** — Long entries invisible once cap hit. Keep hooks to one line.+2. **Priority ordering counterintuitive** — Local beats project beats user beats org. Test full stack.+3. **Extraction timing creates race window** — User can start next turn before background extraction completes.+4. **Derivable content doesn't belong in memory** — Architecture and code patterns are in the repo already.+5. **Concurrent classification is per-call, not per-tool** — Same tool safe for some inputs, unsafe for others.+6. **Permission evaluation has side effects** — Tracks denials, transforms modes, updates state.+7. **Most async work skips "pending" state** — Work units register directly as "running".+8. **Fork children must not fork** — Recursive guard preserves single-level invariant.+9. **Context builders memoized but manually invalidated** — Add invalidation or face staleness.+10. **Hook trust all-or-nothing** — One untrusted hook disables entire extension system.+11. **Eviction requires notification** — Terminal work unit only GC-eligible after parent notified.+12. **Skill listing budgets tight** — Front-load distinctive trigger language, tails get cut.++**Full guide**: [Gotchas](references/gotchas.md) — 15 failure modes with fixes.++## 陷阱(Gotchas)++违反这些非明显原则会导致 bug:++1. **记忆索引上限静默触发** — 条目过长超上限后不可见。保持钩子单行。+2. **优先级顺序反直觉** — 本地胜过项目胜过用户胜过组织。测试完整栈。+3. **提取时序产生竞争窗口** — 用户可在后台提取完成前开始下一轮。+4. **可推导内容不应存入记忆** — 架构和代码模式已在仓库中。+5. **并发分类按调用而非按工具** — 同一工具对某些输入安全,对其他不安全。+6. **权限评估有副作用** — 跟踪拒绝、转换模式、更新状态。+7. **大多数异步工作跳过"pending"状态** — 工作单元直接注册为"运行中"。+8. **Fork 子节点不能 Fork** — 递归防护保持单层不变量。+9. **上下文构建器缓存但手动失效** — 添加失效或面对过时。+10. **Hook 信任全有或全无** — 一个不可信 hook 禁用整个扩展系统。+11. **驱逐需要通知** — 终端工作单元仅在父节点通知后可 GC。+12. **Skill 列表预算紧张** — 前置独特触发语言,尾部被截断。++**完整指南**:[陷阱](references/gotchas.md) — 15 种失败模式及修复方法。++---++## When to Use This Skill++Use this skill when:++- User says "I need to set up AGENTS.md for my project"+- User wants to improve their agent's reliability+- User is experiencing agent failures, lost context, or broken work+- User asks "how do I make my agent work better?"+- User wants to benchmark harness effectiveness+- User needs templates for harness files+- User is following the Learn Harness Engineering course++---++## Communication Style++- Explain harness concepts in practical terms (kitchen analogy works well)+- Focus on measurable outcomes, not theoretical perfection+- Start minimal, add structure as needed+- Show before/after comparisons to build confidence+- Acknowledge tradeoffs (more structure = more reliability but more upfront work)++---++## Getting Started++### If the user is new to harness engineering:++1. **Start with assessment**: Run the five-tuple assessment on their current setup+2. **Pick lowest-scoring subsystem**: Focus improvement efforts there first+3. **Create minimal viable harness**: AGENTS.md + init.sh + feature_list.json+4. **Test with real task**: Measure before/after improvement++### If the user is experienced:++1. **Ask what specific problem**: Don't assume — let them describe the pain point+2. **Understand harness maturity**: What exists already? What's working?+3. **Design targeted improvements**: Use reference patterns for guidance+4. **Optionally run benchmarks**: Quantify impact with before/after comparison++---++## When NOT to Use This Skill++This skill is about the **harness** around an agent, not:+- Prompt engineering or system prompt design+- Model selection or fine-tuning+- Generic software architecture (MVC, microservices)+- Chat UIs or conversational interfaces+- LLM API integration basics++If your question is about the model itself rather than the system around it, this skill does not apply.++---++## Further Resources++- [Learn Harness Engineering Documentation](https://walkinglabs.github.io/learn-harness-engineering/)+- [OpenAI: Harness Engineering](https://openai.com/index/harness-engineering/)+- [Anthropic: Effective Harnesses for Long-Running Agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents/)+- [Anthropic: Harness Design for Long-Running Apps](https://www.anthropic.com/engineering/harness-design-long-running-apps)+- [Awesome Harness Engineering](https://github.com/walkinglabs/awesome-harness-engineering)+- [Agentic Harness Patterns Skill](https://github.com/keli-wen/agentic-harness-patterns-skill) — Reference implementation for pattern extraction++---++## Further Resources++- [Learn Harness Engineering Documentation](https://walkinglabs.github.io/learn-harness-engineering/)+- [OpenAI: Harness Engineering](https://openai.com/index/harness-engineering/)+- [Anthropic: Effective Harnesses for Long-Running Agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents/)+- [Awesome Harness Engineering](https://github.com/walkinglabs/awesome-harness-engineering)diff --git a/.opencode/skills/harness-creator/SKILL.md.en b/.opencode/skills/harness-creator/SKILL.md.ennew file mode 100644index 0000000..8767b02--- /dev/null+++ b/.opencode/skills/harness-creator/SKILL.md.en@@ -0,0 +1,393 @@+---+name: harness-creator+description: >-+ Harness engineering for AI coding agents — five subsystems, memory persistence,+ session continuity, verification workflows, scope control, lifecycle management.+when_to_use: >-+ Use whenever: building harness from scratch, improving agent reliability, agent forgets+ between sessions, agent overreach or scope creep, broken tests after agent work,+ multi-session continuity needed, verification gaps, audit harness quality, benchmark+ effectiveness, create AGENTS.md/CLAUDE.md, design feature tracking, session handoff.+license: MIT+---++# Harness Creator++Production harness engineering for AI coding agents.++**For:** Engineers building or extending coding-agent runtimes, custom agents, multi-session workflows, or anyone who wants their agent to work reliably across sessions.++**Not for:** Prompt engineering, model selection, generic software architecture, or one-off agent tasks.++All principles are grounded in the Learn Harness Engineering framework and production agent runtime decisions.++---++## Choose Your Problem++| If you want to... | Read |+|---|---|+| Make the agent remember corrections and project rules between sessions | [Memory Persistence](references/memory-persistence-pattern.md) |+| Package reusable workflows and domain knowledge | [Skill Runtime](references/skill-runtime-pattern.md) |+| Let the agent work powerfully but not dangerously | [Tool Registry & Safety](references/tool-registry-pattern.md) |+| Give the agent the right context at the right cost | [Context Engineering](references/context-engineering-pattern.md) |+| Split work across multiple agents without chaos | [Multi-agent Coordination](references/multi-agent-pattern.md) |+| Extend behavior with hooks, background tasks, startup logic | [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) |+| Build the complete 5-subsystem harness | [Five Subsystems Guide](#the-five-subsystem-harness-framework) |++**Before you start building:** Read the [Gotchas](#gotchas) — these are the non-obvious failure modes that cost the most time.++---++## The Five-Subsystem Harness Framework++**Every harness consists of five subsystems:**++1. **Instructions (Recipe Shelf)**: AGENTS.md, CLAUDE.md, docs/ hierarchy+2. **State (Prep Station)**: feature_list.json, progress.md, session-handoff.md+3. **Verification (Quality Check Window)**: Verification commands, test suites, type checks+4. **Scope (Task Boundaries)**: One-feature-at-a-time policies, definition of done+5. **Lifecycle (Session Management)**: init.sh, clean-state checklists, handoff procedures++**When creating or improving a harness, systematically address each subsystem.**++---++## Creating a Harness++### Phase 1: Context Gathering++Start by understanding the user's situation:++1. **What project is this for?** (tech stack, size, complexity)+2. **What agent tool are they using?** (Claude Code, Codex, Cursor, etc.)+3. **What exists already?** (any AGENTS.md, progress tracking, verification?)+4. **What problems are they experiencing?** (agent overreach, lost context, broken tests?)+5. **What's the team's tolerance for structure?** (minimal vs. comprehensive)++If the user hasn't provided this context, ask before proceeding.++### Phase 2: Harness Assessment (Existing Projects)++If the user has an existing harness, assess it using the five-tuple framework:++For each subsystem, score 1-5:+- **5**: Exemplary, documented, consistently followed+- **4**: Good, mostly complete, occasional gaps+- **3**: Adequate, covers basics, missing polish+- **2**: Weak, incomplete, inconsistently applied+- **1**: Missing or actively harmful++Identify the lowest-scoring subsystem — that's the bottleneck. Focus improvement efforts there first.++### Phase 3: Design++Based on the assessment, design the harness components:++**Instructions:**+- Create a short AGENTS.md (~50-100 lines) as the routing layer+- Link to detailed docs in docs/ directory (ARCHITECTURE.md, PRODUCT.md, etc.)+- Define startup workflow: what the agent reads before coding++**State:**+- Create feature_list.json with feature definitions and status tracking+- Create or update progress.md for session continuity+- Design session-handoff.md template if needed++**Verification:**+- List explicit verification commands in AGENTS.md+- Ensure init.sh runs verification+- Design quality score tracking if appropriate++**Scope:**+- Define one-feature-at-a-time policy+- Document feature dependencies+- Create definition of done checklist++**Lifecycle:**+- Create init.sh for initialization+- Design clean-state checklist+- Document session handoff procedure++### Phase 4: Implementation++Create the harness files. Use bundled scripts where available:++```bash+# Use bundled scripts from scripts/ directory+# (See scripts/ section for available tools)+```++### Phase 5: Testing and Benchmarking++Test the harness with real agent sessions:++1. **Baseline**: Run a representative task without the harness+2. **With Harness**: Run the same task with the harness+3. **Measure**: Success rate, time, token usage, rework+4. **Compare**: Quantify the improvement++For rigorous benchmarking, see the "Running Benchmarks" section below.++---++## Harness File Templates++### AGENTS.md Structure++A minimal AGENTS.md should include:++```markdown+# AGENTS.md++[One-sentence project purpose]++## Startup Workflow++Before writing code:+1. [Step 1: e.g., Read this file]+2. [Step 2: e.g., Read ARCHITECTURE.md]+3. [Step 3: e.g., Run ./init.sh]+4. [Step 4: e.g., Read feature_list.json]++## Working Rules++- [Rule 1: e.g., One feature at a time]+- [Rule 2: e.g., Verification required before claiming done]+- [Rule 3: e.g., Update progress before ending session]++## Required Artifacts++- `feature_list.json`: Feature state tracker+- `progress.md`: Session continuity log+- `init.sh`: Standard startup and verification++## Definition of Done++A feature is done when:+- [ ] Implementation complete+- [ ] Verification passed+- [ ] Evidence recorded+- [ ] Repository restartable++## End of Session++Before ending:+1. Update progress.md+2. Update feature_list.json+3. Record blockers/risks+4. Commit with descriptive message+5. Leave clean restart path+```++### feature_list.json Structure++```json+{+ "features": [+ {+ "id": "feat-001",+ "name": "Document Import",+ "description": "Allow users to import PDF and TXT documents",+ "dependencies": [],+ "status": "done",+ "evidence": "tests pass, manual verification on 2024-01-15"+ },+ {+ "id": "feat-002",+ "name": "Document Chunking",+ "description": "Split documents into ~500 char chunks with metadata",+ "dependencies": ["feat-001"],+ "status": "in-progress",+ "evidence": ""+ }+ ]+}+```++### init.sh Structure++```bash+#!/bin/bash+set -e++echo "=== Installing dependencies ==="+npm install++echo "=== Running type check ==="+npm run check++echo "=== Running tests ==="+npm test++echo "=== Building application ==="+npm run build++echo "=== Verification complete ==="+```++---++## Running Benchmarks++To measure harness effectiveness:++### Step 1: Define Representative Tasks++Pick 2-3 tasks that are:+- Real work the user would actually do+- Challenging enough to fail without proper harness+- Verifiable (clear success criteria)++### Step 2: Run Comparative Sessions++For each task:+- **Without Harness**: Run the task on a clean repo copy+- **With Harness**: Run the same task with the harness in place++Record:+- Success/failure+- Time taken+- Token usage+- Rework required+- Session restarts needed++### Step 3: Aggregate Results++Calculate:+- Success rate improvement+- Time efficiency change+- Token efficiency change+- Qualitative feedback++### Step 4: Iterate++Use results to identify:+- Which harness components add most value+- Which components are over-engineered+- Where to focus improvement efforts++---++## Bundled Resources++### References (Deep-Dive Patterns)++| Document | Covers |+|----------|--------|+| [Memory Persistence](references/memory-persistence-pattern.md) | Four-level instruction hierarchy, auto-memory taxonomy, background extraction |+| [Context Engineering](references/context-engineering-pattern.md) | Select / Compress / Isolate / Write operations, budget management |+| [Tool Registry](references/tool-registry-pattern.md) | Fail-closed registration, per-call concurrency, permission pipeline |+| [Multi-Agent](references/multi-agent-pattern.md) | Coordinator / Fork / Swarm patterns, context sharing |+| [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) | Hook system, long-running tasks, dependency-ordered init |+| [Gotchas](references/gotchas.md) | 15 non-obvious failure modes with fixes |++### Templates++- `templates/agents.md` — AGENTS.md / CLAUDE.md skeleton+- `templates/feature-list.json` — Feature state tracker+- `templates/init.sh` — Standard initialization script+- `templates/progress.md` — Session progress log+- `templates/session-handoff.md` — Session handoff template++### Scripts (Optional)++- `scripts/create-harness.ts` — Generate harness files from templates+- `scripts/validate-harness.ts` — Check harness completeness+- `scripts/run-benchmark.ts` — Execute harness effectiveness comparison++---++## Gotchas++Non-obvious principles that will cause bugs if you violate them:++1. **Memory index caps fire silently** — Long entries invisible once cap hit. Keep hooks to one line.+2. **Priority ordering counterintuitive** — Local beats project beats user beats org. Test full stack.+3. **Extraction timing creates race window** — User can start next turn before background extraction completes.+4. **Derivable content doesn't belong in memory** — Architecture and code patterns are in the repo already.+5. **Concurrent classification is per-call, not per-tool** — Same tool safe for some inputs, unsafe for others.+6. **Permission evaluation has side effects** — Tracks denials, transforms modes, updates state.+7. **Most async work skips "pending" state** — Work units register directly as "running".+8. **Fork children must not fork** — Recursive guard preserves single-level invariant.+9. **Context builders memoized but manually invalidated** — Add invalidation or face staleness.+10. **Hook trust all-or-nothing** — One untrusted hook disables entire extension system.+11. **Eviction requires notification** — Terminal work unit only GC-eligible after parent notified.+12. **Skill listing budgets tight** — Front-load distinctive trigger language, tails get cut.++**Full guide**: [Gotchas](references/gotchas.md) — 15 failure modes with fixes.++---++## When to Use This Skill++Use this skill when:++- User says "I need to set up AGENTS.md for my project"+- User wants to improve their agent's reliability+- User is experiencing agent failures, lost context, or broken work+- User asks "how do I make my agent work better?"+- User wants to benchmark harness effectiveness+- User needs templates for harness files+- User is following the Learn Harness Engineering course++---++## Communication Style++- Explain harness concepts in practical terms (kitchen analogy works well)+- Focus on measurable outcomes, not theoretical perfection+- Start minimal, add structure as needed+- Show before/after comparisons to build confidence+- Acknowledge tradeoffs (more structure = more reliability but more upfront work)++---++## Getting Started++### If the user is new to harness engineering:++1. **Start with assessment**: Run the five-tuple assessment on their current setup+2. **Pick lowest-scoring subsystem**: Focus improvement efforts there first+3. **Create minimal viable harness**: AGENTS.md + init.sh + feature_list.json+4. **Test with real task**: Measure before/after improvement++### If the user is experienced:++1. **Ask what specific problem**: Don't assume — let them describe the pain point+2. **Understand harness maturity**: What exists already? What's working?+3. **Design targeted improvements**: Use reference patterns for guidance+4. **Optionally run benchmarks**: Quantify impact with before/after comparison++---++## When NOT to Use This Skill++This skill is about the **harness** around an agent, not:+- Prompt engineering or system prompt design+- Model selection or fine-tuning+- Generic software architecture (MVC, microservices)+- Chat UIs or conversational interfaces+- LLM API integration basics++If your question is about the model itself rather than the system around it, this skill does not apply.++---++## Further Resources++- [Learn Harness Engineering Documentation](https://walkinglabs.github.io/learn-harness-engineering/)+- [OpenAI: Harness Engineering](https://openai.com/index/harness-engineering/)+- [Anthropic: Effective Harnesses for Long-Running Agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents/)+- [Anthropic: Harness Design for Long-Running Apps](https://www.anthropic.com/engineering/harness-design-long-running-apps)+- [Awesome Harness Engineering](https://github.com/walkinglabs/awesome-harness-engineering)+- [Agentic Harness Patterns Skill](https://github.com/keli-wen/agentic-harness-patterns-skill) — Reference implementation for pattern extraction++---++## Further Resources++- [Learn Harness Engineering Documentation](https://walkinglabs.github.io/learn-harness-engineering/)+- [OpenAI: Harness Engineering](https://openai.com/index/harness-engineering/)+- [Anthropic: Effective Harnesses for Long-Running Agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents/)+- [Awesome Harness Engineering](https://github.com/walkinglabs/awesome-harness-engineering)diff --git a/.opencode/skills/harness-creator/evals/evals.json b/.opencode/skills/harness-creator/evals/evals.jsonnew file mode 100644index 0000000..af67269--- /dev/null+++ b/.opencode/skills/harness-creator/evals/evals.json@@ -0,0 +1,71 @@+{+ "skill_name": "harness-creator",+ "evals": [+ {+ "id": 1,+ "name": "Minimal Harness Creation",+ "prompt": "I have a new TypeScript + React project with no agent setup. Create a minimal harness that makes my agent reliable for single-feature development.",+ "expected_output": "AGENTS.md (~50-100 lines), feature_list.json with 3-5 placeholder features, init.sh with verification commands",+ "files": [],+ "expectations": [+ "AGENTS.md includes startup workflow (read files, run init, check feature list)",+ "AGENTS.md includes one-feature-at-a-time policy",+ "feature_list.json has valid JSON with id, name, description, status fields",+ "init.sh runs npm install, type check, and test commands",+ "All files are in project root directory"+ ]+ },+ {+ "id": 2,+ "name": "Session Continuity Setup",+ "prompt": "My agent forgets everything between sessions. I need persistent memory and session handoff so it can work on multi-day features.",+ "expected_output": "progress.md template, session-handoff.md structure, memory directory setup instructions",+ "files": [],+ "expectations": [+ "progress.md includes sections: current state, what's done, what's in progress, blockers, next session should",+ "session-handoff.md includes: what was accomplished, what remains, blockers/decisions, files modified",+ "Instructions for memory directory creation (.claude/memory/ or similar)",+ "Two-step save invariant explained (topic file then index)"+ ]+ },+ {+ "id": 3,+ "name": "Harness Assessment",+ "prompt": "I have an existing AGENTS.md file but my agent still breaks things. Assess my harness and tell me what to improve first.",+ "expected_output": "Five-subsystem assessment with scores 1-5 for each, bottleneck identified, prioritized improvement plan",+ "files": ["existing-AGENTS.md"],+ "expectations": [+ "Assessment covers all 5 subsystems: Instructions, State, Verification, Scope, Lifecycle",+ "Each subsystem scored 1-5 with justification",+ "Lowest-scoring subsystem identified as bottleneck",+ "Prioritized improvement plan with 2-3 concrete next steps"+ ]+ },+ {+ "id": 4,+ "name": "Verification Workflow Design",+ "prompt": "My agent says 'done' but the tests fail. Design a verification workflow that forces the agent to actually verify before claiming completion.",+ "expected_output": "Verification commands list, AGENTS.md section updates, optional quality score tracking",+ "files": [],+ "expectations": [+ "Explicit verification commands listed (tests, lint, type-check, build)",+ "AGENTS.md includes 'Definition of Done' section with verification requirement",+ "End-of-session checklist includes verification evidence recording",+ "Optional: quality score tracking mechanism described"+ ]+ },+ {+ "id": 5,+ "name": "Memory Taxonomy Design",+ "prompt": "I want my agent to remember project conventions and user preferences across sessions. Design a memory taxonomy and tell me what belongs where.",+ "expected_output": "Memory layer definitions, type taxonomy for auto-memory, what to save vs what to skip",+ "files": [],+ "expectations": [+ "Instruction memory defined (human-curated, version-controlled)",+ "Auto-memory defined (agent-written, persistent)",+ "Type taxonomy with 3-4 types (e.g., user/feedback/project/reference)",+ "Clear guidance on what NOT to save (derivable content)"+ ]+ }+ ]+}diff --git a/.opencode/skills/harness-creator/metadata.json b/.opencode/skills/harness-creator/metadata.jsonnew file mode 100644index 0000000..8a16b6e--- /dev/null+++ b/.opencode/skills/harness-creator/metadata.json@@ -0,0 +1,55 @@+{+ "name": "harness-creator",+ "version": "1.0.0",+ "description": "Harness engineering for AI coding agents — five subsystems, memory persistence, session continuity, verification workflows, scope control, lifecycle management.",+ "license": "MIT",+ "author": "Learn Harness Engineering",+ "repository": "https://github.com/walkinglabs/learn-harness-engineering",+ "languages": ["en", "zh"],+ "compatibility": {+ "minCliVersion": "0.1.0",+ "agents": [+ "claude-code",+ "codex-cli",+ "cursor",+ "windsurf",+ "generic"+ ]+ },+ "entryPoint": "SKILL.md",+ "references": [+ "references/memory-persistence-pattern.md",+ "references/context-engineering-pattern.md",+ "references/tool-registry-pattern.md",+ "references/multi-agent-pattern.md",+ "references/lifecycle-bootstrap-pattern.md"+ ],+ "triggers": [+ "harness engineering",+ "agent reliability",+ "session continuity",+ "memory persistence",+ "verification workflow",+ "scope control",+ "lifecycle management",+ "AGENTS.md",+ "CLAUDE.md",+ "feature tracking",+ "session handoff",+ "multi-agent coordination",+ "context engineering",+ "tool safety",+ "permission pipeline"+ ],+ "bundled": {+ "scripts": [],+ "assets": [],+ "templates": [+ "templates/agents.md",+ "templates/feature-list.json",+ "templates/init.sh",+ "templates/progress.md",+ "templates/session-handoff.md"+ ]+ }+}diff --git a/.opencode/skills/harness-creator/references/context-engineering-pattern.md b/.opencode/skills/harness-creator/references/context-engineering-pattern.mdnew file mode 100644index 0000000..6bf6e3d--- /dev/null+++ b/.opencode/skills/harness-creator/references/context-engineering-pattern.md@@ -0,0 +1,150 @@+# Context Engineering Pattern++## Problem++Agents fail when context is managed poorly:+- **Too much context** → Session startup is slow, token costs explode, model gets lost in details+- **Too little context** → Agent makes wrong assumptions, reinvents wheels, violates conventions+- **Wrong context** → Agent focuses on low-level details, misses architectural constraints++Context is not a dump. It's a budget that must be managed with explicit operations.++## Golden Rules++### Four Context Operations++Every token in the window should earn its place through one of four operations:++1. **SELECT** — Load context just-in-time, not all-at-once+2. **WRITE** — Agent writes back to persistent storage (memory, state, rules)+3. **COMPRESS** — Reactive compaction of older turns mid-session+4. **ISOLATE** — Delegated work must not pollute parent context++### Progressive Disclosure++Three-tier loading:++```+Tier 1: Metadata (always present, cheap)+ → Feature list, memory index, session status++Tier 2: Instructions (loaded on activation)+ → AGENTS.md, skill bodies, style guides++Tier 3: Resources (loaded on demand)+ → Architecture docs, API references, examples+```++### Memoize Expensive Builders, Invalidate Explicitly++Context builders (e.g., "load all recent git commits") should be memoized to avoid redundant work, but **must** be invalidated at known mutation points — not reactively. Every mutation point must clear its corresponding cache.++## When To Use++- Agent performance degrades in long sessions+- Startup is slow due to eager context loading+- Delegated work pollutes the parent context+- Token costs are unpredictable++## Tradeoffs++| Decision | Benefit | Cost |+|---|---|---|+| JIT loading | Fast startup, low idle cost | Agent can't reason about skills until activated |+| Hard caps per block | Predictable token budget | May truncate useful context |+| Manual cache invalidation | No reactive staleness | Developer must add invalidation at each mutation |+| Isolation for delegation | Clean parent context | Child can't see parent's accumulated context |++## Implementation Patterns++### Select Pattern++```markdown+## Startup Context (Loaded Immediately)++- Repository root path+- Tech stack (one line)+- Active feature ID from feature_list.json++## On-Demand Context (Loaded When Triggered)++- Skill: Read when skill activates+- Architecture docs: Read when implementing new feature+- API reference: Read when calling external services+```++**Key moves:**+- Audit current context cost per turn+- Apply hard caps to every variable-length block+- Add truncation recovery pointers ("call list_files for full output")++### Compress Pattern++Long sessions exhaust the window. Reactive compaction:++1. **Trigger**: Context usage exceeds threshold (e.g., 80%)+2. **Summarize**: Older turns (first 50% by token count)+3. **Preserve**: Recent context (last 20% of turns)+4. **Label**: Mark snapshot as "compacted at turn N"++```markdown+## Session Summary (Turns 1-15, compacted)++**Goal**: Implement Q&A feature with citations+**Decisions made**:+- Use streaming response for UX+- Citation format: [doc:chunk] inline references+**Key files created**:+- src/services/QaService.ts+- src/shared/types.ts (extended with QaResult)+```++### Isolate Pattern++Delegated work must not pollute parent context:++| Pattern | Context Sharing | Best For |+|---|---|---|+| **Coordinator** (zero inheritance) | None — workers start fresh | Complex multi-phase tasks |+| **Fork** (full inheritance) | Full — single-level only | Quick parallel splits |+| **Swarm** (peer-to-peer) | Shared task list | Long-running independent work |++**Key constraint**: Fork is single-level only — recursive forks multiply context cost exponentially.++## Gotchas++1. **Most async work skips "pending" state** — work units register directly as "running"+2. **Context builders are memoized but manually invalidated** — add invalidation or face staleness+3. **Truncation is silent until it fires** — hard caps enforced at read time+4. **Isolation boundary must be enforced at call time** — don't just remove tools from prompt++## Related Patterns++- [Memory Persistence](memory-persistence-pattern.md) — How memory layers interact with context+- [Multi-agent Coordination](multi-agent-pattern.md) — Context sharing across agents++## Template: Context Budget++```markdown+## Context Budget (Session)++| Category | Budget | Current | Status |+|----------|--------|---------|--------|+| System prompt | 2,000 | 1,850 | ✓ |+| Instruction files | 3,000 | 2,400 | ✓ |+| Memory index | 1,000 | 600 | ✓ |+| Session history | 10,000 | 4,200 | ✓ |+| Working context | 15,000 | 3,100 | ✓ |+| **Total** | **31,000** | **12,150** | 39% used |++**Compaction trigger**: 80% (24,800 tokens)+**Next action**: Trigger compaction at 24,800 tokens+```++## Evidence++Context engineering patterns are observed in production agent runtimes where:+- Context budgets are explicit, not implicit+- Progressive disclosure reduces startup latency by 60-80%+- Manual cache invalidation prevents subtle staleness bugs+- Isolation patterns enable reliable multi-agent coordinationdiff --git a/.opencode/skills/harness-creator/references/gotchas.md b/.opencode/skills/harness-creator/references/gotchas.mdnew file mode 100644index 0000000..c8cd51b--- /dev/null+++ b/.opencode/skills/harness-creator/references/gotchas.md@@ -0,0 +1,209 @@+# Gotchas — Harness Engineering Failure Modes++Non-obvious principles that will cause bugs if you violate them.++---++## 1. Memory Index Caps Fire Silently++**Symptom**: Recent memories "disappear" without error.++**Cause**: Index has hard caps (e.g., 200 lines / 25KB) enforced at read time. Long entries (multi-sentence summaries) hit byte cap while staying under line cap.++**Fix**: Keep index entries to one-line hooks. Put detail in topic files.++```markdown+✓ Good: "Use bun, not npm - user preference 2024-01-15"+✗ Bad: "The user prefers bun over npm because it's faster. This was discussed on 2024-01-15 when the user said 'use bun not npm' and I updated the package.json accordingly..."+```++---++## 2. Priority Ordering is Counterintuitive++**Symptom**: Global rule silently overridden by local file.++**Cause**: Local overrides beat project rules, which beat user rules, which beat org rules. If you inject at user level expecting it to dominate, a local override file in project root wins.++**Fix**: Test with full instruction-file stack present:++```bash+# Test priority ordering+cat ~/.claude/CLAUDE.md # User level+cat ./CLAUDE.md # Project level+cat ./CLAUDE.local.md # Local override (WINS)+```++---++## 3. Extraction Timing Creates Race Window++**Symptom**: Background extractor writes memory, but user starts next turn before extraction completes.++**Cause**: Extraction fires at end of response. User can send message before extraction finishes.++**Fix**: Coalesce concurrent extraction requests. Advance cursor only after successful run. Failed extraction means those messages reconsidered next time.++---++## 4. Derivable Content Doesn't Belong in Memory++**Symptom**: Memory index fills with architecture details that stale quickly.++**Cause**: Agent saves what's derivable from codebase (architecture, code patterns, version history).++**Fix**: Exclude derivable content by design. Type taxonomy should forbid saving what's in the repo already.++---++## 5. Concurrent Classification is Per-Call, Not Per-Tool++**Symptom**: Tool marked "concurrent-safe" causes race conditions.++**Cause**: Same tool can be safe for some inputs and unsafe for others. Don't assume tool's concurrency behavior is static.++**Fix**: Classify each call at runtime:++```typescript+// Don't do this:+toolRegistry.register('shell', { concurrentSafe: false });++// Do this:+function isCallConcurrentSafe(call: ToolCall): boolean {+ if (call.args.command.startsWith('rm -rf')) return false;+ if (call.args.command.startsWith('cat')) return true;+ // ...runtime classification+}+```++---++## 6. Permission Evaluation Has Side Effects++**Symptom**: Permission check changes behavior on subsequent calls.++**Cause**: Permission evaluator tracks denials, transforms modes, updates state as side effect. Not a pure lookup function.++**Fix**: Don't cache permission results across calls. Re-evaluate each call fresh.++---++## 7. Most Async Work Skips "Pending" State++**Symptom**: UI shows "pending" but work unit never enters that state.++**Cause**: Work units register directly as "running" in practice. "Pending" exists in state machine but rarely used.++**Fix**: Don't build UI that assumes every work unit starts pending.++---++## 8. Fork Children Must Not Fork++**Symptom**: Context cost explodes exponentially.++**Cause**: Recursive forks multiply context: parent + child1 + child2 + grandchildren...++**Fix**: Enforce single-level invariant. Keep fork tool in child's pool (for prompt cache sharing) but block at call time.++---++## 9. Context Builders are Memoized but Manually Invalidated++**Symptom**: Model sees stale data for entire session.++**Cause**: Context builder cached at startup, but mutation doesn't clear cache.++**Fix**: Every mutation point must explicitly clear its corresponding cache:++```typescript+// Example: Cache invalidation at mutation point+async function editFile(path: string, content: string) {+ await writeFile(path, content);+ context.cache.invalidate(`file:${path}`); // MUST invalidate+}+```++---++## 10. Hook Trust is All-or-Nothing++**Symptom**: Entire extension system disabled because one hook untrusted.++**Cause**: If workspace untrusted, all hooks skip — not just suspicious ones.++**Fix**: Design hooks with trust gate at dispatch point. Don't attempt per-hook trust evaluation.++---++## 11. Eviction Requires Notification++**Symptom**: Parent can never read work unit result.++**Cause**: Work unit evicted before parent notified of completion. Race condition: parent tries to read result that's already GC'd.++**Fix**: Two-phase eviction:+1. Clean disk output at terminal state (eager)+2. Clean in-memory record after parent notified (lazy)++---++## 12. Skill Listing Budgets Are Tight++**Symptom**: Skill description truncated, can't trigger properly.++**Cause**: Skill descriptions concatenated and capped per entry (~150 chars). Front-loaded trigger language gets priority.++**Fix**: Front-load distinctive trigger language:++```markdown+✓ Good: "harness-patterns: Memory, permissions, context engineering, multi-agent"+✗ Bad: "A comprehensive skill for understanding and implementing various patterns related to AI agent harnesses and runtime systems..."+```++---++## 13. Default Tool Permission is "Allow"++**Symptom**: Tool bypasses expected gate.++**Cause**: Tools without custom permission logic delegate entirely to rule-based system. Default is "allow" unless configured otherwise.++**Fix**: Override default for sensitive tools:++```typescript+registry.register('shell', {+ defaultPermission: 'ask', // NOT 'allow'+ // ...+});+```++---++## 14. Team Memory Requires Auto-Memory Enabled++**Symptom**: Team-shared memory doesn't work even when configured.++**Cause**: Team memory builds on same directory/index infrastructure as auto-memory. Disabling auto-memory (via env var or settings) also disables team memory.++**Fix**: Ensure auto-memory enabled before enabling team memory. Check both feature gate and enablement check.++---++## 15. Orphaned Topic Files Accumulate++**Symptom**: Disk space fills with `.claude/memory/topics/` files.++**Cause**: Two-step save (topic file then index). Crash between steps leaves orphaned topic file.++**Fix**: Periodic sweep deletes topic files not referenced by index. Orphans don't corrupt index but consume disk space.++---++## Related Reading++- [Memory Persistence Pattern](memory-persistence-pattern.md) — Gotchas #1, #3, #4, #15+- [Tool Registry Pattern](tool-registry-pattern.md) — Gotchas #5, #6, #13+- [Multi-agent Pattern](multi-agent-pattern.md) — Gotchas #8, #11+- [Context Engineering Pattern](context-engineering-pattern.md) — Gotchas #9+- [Lifecycle Pattern](lifecycle-bootstrap-pattern.md) — Gotchas #10, #14diff --git a/.opencode/skills/harness-creator/references/lifecycle-bootstrap-pattern.md b/.opencode/skills/harness-creator/references/lifecycle-bootstrap-pattern.mdnew file mode 100644index 0000000..3fdb127--- /dev/null+++ b/.opencode/skills/harness-creator/references/lifecycle-bootstrap-pattern.md@@ -0,0 +1,264 @@+# Lifecycle and Bootstrap Pattern++## Problem++Agent runtimes need extensibility without compromising safety:++- **Hooks** — Extend behavior at lifecycle moments (pre/post tool execution, session start/end)+- **Background tasks** — Track long-running work without blocking the main agent+- **Bootstrap** — Structure initialization across multiple entry modes (CLI, server, SDK)++But uncontrolled extensibility creates:+- Security holes from untrusted hooks+- Resource leaks from tasks that never complete+- Race conditions in initialization++## Golden Rules++### Hook Trust is All-or-Nothing++If the workspace is untrusted, **all hooks skip** — not just suspicious ones. Session-scoped hooks are ephemeral and cleaned on session end.++```typescript+// Example: Hook dispatch with trust gate+async function dispatchHook(+ hookType: HookType,+ context: HookContext+): Promise<HookResult[]> {++ // Trust gate: if workspace untrusted, skip ALL hooks+ if (!context.trustBoundary.crossed) {+ logger.warn('Untrusted workspace, skipping hooks');+ return [];+ }++ // Session-scoped hooks ephemeral — cleanup on session end+ const sessionHooks = context.hooks.getByScope('session');+ const projectHooks = context.hooks.getByScope('project');++ return await Promise.all([+ ...sessionHooks.map(h => h.execute(context)),+ ...projectHooks.map(h => h.execute(context)),+ ]);+}+```++### Long-Running Work: Typed State Machines with Two-Phase Eviction++Each work unit gets:+1. **Typed, prefixed ID** (e.g., `extractor-001`, `benchmark-002`)+2. **Strict lifecycle** (running → completed | failed | killed)+3. **Disk-backed output** (not just in-memory)++Eviction is two-phase:+1. **Disk output** cleaned eagerly at terminal state+2. **In-memory records** cleaned lazily after parent notified++### Bootstrap: Dependency-Ordered, Memoized Stages++Multiple entry modes (CLI, server, SDK) share the same bootstrap path:++```+Stage 1: Create minimal context (no trust required)+ ↓+Stage 2: Load tools (read-only safe)+ ↓+Stage 3: Trust boundary crossed (user grants consent)+ ↓+Stage 4: Load security-sensitive subsystems (telemetry, secret env vars)+```++**Critical inflection**: Security-sensitive subsystems must not activate before trust is established.++## When To Use++- You need to extend agent behavior without modifying core code+- You need to track long-running background work+- You need structured initialization across multiple entry modes+- You need hooks at lifecycle moments (pre/post tool, session start/end)++## Tradeoffs++| Decision | Benefit | Cost |+|---|---|---|+| All-or-nothing hook trust | Simple security boundary | One untrusted hook disables entire extension system |+| Disk-backed task output | Memory constant regardless of concurrent work | I/O latency proportional to work units |+| Dependency-ordered bootstrap | Multiple entry modes share path | Initial startup sequential (can't parallelize stages) |+| Memoized stages | Re-init is fast | Must carefully invalidate memoization on config change |++## Implementation Patterns++### Hook Lifecycle++Six hook types dispatched at defined moments:++```typescript+interface HookRegistry {+ // Session lifecycle+ onSessionStart: (context: SessionContext) => Promise<void>;+ onSessionEnd: (context: SessionContext) => Promise<void>;++ // Tool execution+ preToolExecute: (context: ToolContext) => Promise<ToolContext>;+ postToolExecute: (context: ToolResult) => Promise<ToolResult>;++ // Prompt submission+ prePromptSubmit: (context: PromptContext) => Promise<PromptContext>;+ postPromptSubmit: (context: ResponseContext) => Promise<ResponseContext>;+}++// Usage: Register hooks via config+// /update-config hooks.preToolExecute = "scripts/audit-tool-call.js"+```++### Long-Running Task Tracking++```typescript+interface TaskRegistry {+ // Typed prefixed IDs+ registerWork(+ type: 'extraction' | 'benchmark' | 'indexing',+ outputType: 'json' | 'text' | 'file'+ ): string; // Returns typed ID: `extraction-001`++ // Strict state machine+ updateState(+ taskId: string,+ state: 'running' | 'completed' | 'failed' | 'killed',+ output?: any+ ): void;++ // Two-phase eviction+ evictTask(taskId: string): void;+ // 1. Clean disk output (eager, at terminal state)+ // 2. Clean in-memory record (lazy, after parent notified)+}+```++### Bootstrap Sequence++```typescript+// Example: Dependency-ordered initialization+class AgentBootstrap {+ private stages = new Map<string, Stage>();+ private memoizedCallers = new Map<string, any>();++ async bootstrap(entryMode: 'cli' | 'server' | 'sdk'): Promise<AgentContext> {++ // Stage 1: Minimal context (no trust required)+ await this.runStage('minimal-context', async () => {+ return {+ cwd: process.cwd(),+ entryMode,+ trustBoundary: { crossed: false },+ };+ });++ // Stage 2: Load tools (read-only safe)+ await this.runStage('load-tools', async (context) => {+ context.tools = await this.loadSafeTools();+ return context;+ });++ // Stage 3: Trust boundary (user grants consent)+ await this.runStage('trust-boundary', async (context) => {+ const consent = await this.requestConsent();+ context.trustBoundary = { crossed: consent };+ return context;+ });++ // Stage 4: Security-sensitive subsystems (requires trust)+ if (context.trustBoundary.crossed) {+ await this.runStage('load-sensitive', async (context) => {+ context.telemetry = await this.loadTelemetry();+ context.secretEnvVars = await this.loadSecrets();+ return context;+ });+ }++ return context;+ }++ private async runStage(+ name: string,+ fn: (context: AgentContext) => Promise<AgentContext>+ ): Promise<void> {+ // Memoized: skip if already run+ if (this.stages.has(name) && this.stages.get(name).complete) {+ return;+ }++ // Run stage+ const stage = { name, complete: false, running: true };+ this.stages.set(name, stage);++ try {+ await fn(this.context);+ stage.complete = true;+ } finally {+ stage.running = false;+ }+ }+}+```++## Gotchas++1. **Hook trust is all-or-nothing** — One untrusted hook disables entire extension system+2. **Most async work skips "pending" state** — Work units register directly as "running"+3. **Eviction requires notification** — Terminal work unit only GC-eligible after parent notified+4. **Fast-path dispatch** — Memoized callers must handle concurrent calls without re-running stages+5. **Hook types must be disjoint** — Don't create overlapping hook scopes++## Related Patterns++- [Tool Registry](tool-registry-pattern.md) — How tools are registered at bootstrap+- [Memory Persistence](memory-persistence-pattern.md) — How memory is loaded at init++## Template: Bootstrap Checklist++Before declaring bootstrap complete:++```markdown+## Bootstrap Verification++### Stage 1: Minimal Context+- [ ] Working directory confirmed+- [ ] Entry mode determined (cli / server / sdk)+- [ ] Trust boundary NOT crossed (no secrets loaded)++### Stage 2: Tools Loaded+- [ ] Read-only tools registered (read, search, glob)+- [ ] Write tools NOT yet registered (edit, shell)+- [ ] Tool permissions set to default (ask / deny)++### Stage 3: Trust Boundary+- [ ] User consent requested (interactive or config flag)+- [ ] Consent recorded in session state+- [ ] Security audit logged++### Stage 4: Sensitive Subsystems+- [ ] Telemetry initialized (if consent given)+- [ ] Secret env vars loaded (if consent given)+- [ ] Write tools registered (edit, shell, exec)+- [ ] Hook system enabled (if workspace trusted)++### Stage 5: Background Tasks+- [ ] Task registry initialized+- [ ] Cleanup handlers registered+- [ ] Drain-on-shutdown configured++## If Any Stage Fails++- Bootstrap halts immediately+- Session remains in safe mode (read-only)+- Error logged with stage name and failure reason+```++## Evidence++Lifecycle and bootstrap patterns are observed in production runtimes where:+- Hook dispatch is all-or-nothing based on workspace trust+- Long-running tasks use typed prefixed IDs and disk-backed output+- Bootstrap is dependency-ordered with memoized stages+- Trust boundary is explicit inflection point for security-sensitive subsystemsdiff --git a/.opencode/skills/harness-creator/references/memory-persistence-pattern.md b/.opencode/skills/harness-creator/references/memory-persistence-pattern.mdnew file mode 100644index 0000000..089e0ac--- /dev/null+++ b/.opencode/skills/harness-creator/references/memory-persistence-pattern.md@@ -0,0 +1,110 @@+# Memory and Persistence Pattern++## Problem++Without persistent memory, an agent loses all user preferences, project context, and behavioral feedback the moment a session ends. Users must repeat corrections every session ("use bun, not npm"), and the agent cannot accumulate the working knowledge that makes it genuinely useful over time.++## Golden Rules++### Separate layers by scope and durability++- **Instruction memory** (human-curated, version-controlled): AGENTS.md, CLAUDE.md, project conventions+- **Auto-memory** (agent-written, persistent): Progress logs, session handoffs, discovered patterns+- **Session extraction** (background-derived): Automatic transcript analysis at session end++### Two-step save invariant++Every memory write is a two-step operation:+1. Write the full content to a dedicated topic file+2. Append a one-line pointer to the index++If the process crashes between steps, the worst outcome is an orphaned topic file — the index remains consistent.++### Local overrides win — always++When the same topic is addressed at multiple scopes, the most-local instruction takes priority:++```+Organization-wide → User-level → Project-level → Local override+ ↓ ↓ ↓ ↓+ sets floor narrows it narrows further final say+```++### The index is bounded always-on context; topic files are on-demand detail++- **Index**: Hard-capped at ~200 lines / 25KB, one line per entry+- **Topic files**: Unlimited detail, loaded on demand++## When To Use++- Your agent persists across sessions and must recall user preferences or project context+- Multiple scopes of instruction coexist and need clear priority ordering+- The agent should learn from sessions without manual curation+- You need background extraction that doesn't block the user++## Tradeoffs++| Decision | Benefit | Cost |+|---|---|---|+| Layered memory | Each scope can be shared, audited, overridden independently | More files to discover at startup |+| Local-wins priority | Users can override without touching shared files | Global rule can be silently overridden |+| Bounded index with on-demand topics | Constant context cost regardless of memory volume | Agent must perform extra retrieval step |+| Background extraction | No latency added to user responses | Race window between extraction and next turn |++## Implementation Patterns++1. **Define memory directory** idempotently at startup (e.g., `.claude/memory/`)+2. **Create index file** with hard caps enforced at read time+3. **Implement two-step save**: topic file first, then index update+4. **Fire background extraction** only after final response with no pending tool calls+5. **Enforce mutual exclusion**: if main agent wrote to memory, skip extraction that turn+6. **Build review mechanism** for cross-layer promotion proposals++## Gotchas++1. **Index truncation is silent until it fires** — keep entries short+2. **Priority ordering is counterintuitive** — local beats project beats user beats org+3. **Extraction timing creates a race window** — user can start next turn before extraction completes+4. **Derivable content doesn't belong in memory** — architecture and code patterns are re-derivable from the codebase+5. **Orphaned topic files accumulate** — periodic cleanup recommended++## Related Patterns++- [Context Engineering](context-engineering-pattern.md) — How to manage context budget across layers+- [Lifecycle & Bootstrap](lifecycle-bootstrap-pattern.md) — How initialization loads memory++## Template: Progress Log Structure++```markdown+# Session Progress Log++## Current State (Last Updated: YYYY-MM-DD HH:MM)++**Active Feature:** feat-003 - Q&A with Citations+**Status:** In Progress (60% complete)++### What's Done+- [x] Document chunking pipeline+- [x] Index data structure+- [ ] Q&A handler (in progress)++### What's In Progress+- Implementing Q&A IPC handler+- Need to decide: streaming vs batch response++### Blockers+- Waiting on decision: citation format (footnotes vs inline)++### Next Session Should+1. Complete Q&A handler+2. Add citation formatting+3. Test end-to-end flow+```++## Evidence++This pattern is grounded in production agent runtimes including Claude Code's memory system, which implements:+- Four-level instruction hierarchy (org/user/project/local)+- Four-type auto-memory taxonomy (user/feedback/project/reference)+- Background session extraction with mutual exclusion+- Team-shared memory as an extension layerdiff --git a/.opencode/skills/harness-creator/references/multi-agent-pattern.md b/.opencode/skills/harness-creator/references/multi-agent-pattern.mdnew file mode 100644index 0000000..36a0aa3--- /dev/null+++ b/.opencode/skills/harness-creator/references/multi-agent-pattern.md@@ -0,0 +1,193 @@+# Multi-Agent Coordination Pattern++## Problem++Single agents hit limits:+- **Context limits** — Can't hold full research + implementation in one session+- **Specialization** — Need separate researchers, implementers, reviewers+- **Parallelism** — Want to explore multiple approaches simultaneously++But multi-agent systems introduce chaos:+- Workers duplicate each other's research+- Coordinators delegate understanding instead of synthesizing+- Context inheritance explodes exponentially++## Golden Rules++### The Coordinator Must Synthesize, Not Delegate Understanding++**Anti-pattern:**+> "Based on your findings, fix the authentication system."++**Pattern:**+> "Research identified 3 auth flows: login, logout, token refresh. Implement ONLY the token refresh handler using the JWT strategy documented in [research output]. Return: implementation diff + test results."++The coordinator (orchestrator) adds value by digesting worker results into precise specs before dispatching implementation.++### Three Delegation Patterns++| Pattern | Context Sharing | Best For | Constraints |+|---------|----------------|----------|-------------|+| **Coordinator** | None — workers start fresh | Complex multi-phase tasks (research → synthesize → implement → verify) | Slowest but safest |+| **Fork** | Full — child inherits parent history | Quick parallel splits sharing loaded context | **Single-level only** — recursive forks multiply context cost |+| **Swarm** | Peer-to-peer via shared task list | Long-running independent workstreams | **Flat roster** — teammates can't spawn other teammates |++### Results Arrive Asynchronously; Fire-and-Forget Registration Returns ID Immediately++```typescript+// Example: Spawn worker, get ID back immediately+const taskId = await coordinator.spawn({+ type: 'research',+ prompt: 'Analyze auth flows...',+ toolFilter: ['read', 'search'], // Restrict tools+});++// Parent can continue working while worker runs+// Results arrive via callback or polling+```++## When To Use++- Task too large for single agent session+- Need parallel exploration (e.g., prototype multiple approaches)+- Want persistent specialized teammates (researcher, implementer, reviewer)+- Complex multi-phase workflows++## Tradeoffs++| Pattern | Speed | Safety | Context Cost |+|---------|-------|--------|--------------|+| **Coordinator** | Slowest | Safest | Lowest (zero inheritance) |+| **Fork** | Fastest | Medium | Highest (full inheritance) |+| **Swarm** | Medium | Medium | Medium (shared state only) |++## Implementation Patterns++### Coordinator Pattern (Recommended for Complex Tasks)++Phased workflow:++```+Phase 1: Research+ ↓ (synthesize findings)+Phase 2: Plan+ ↓ (precise specs)+Phase 3: Implement+ ↓ (verify)+Phase 4: Review+```++```typescript+// Example: Coordinator workflow+const research = await coordinator.spawn({+ role: 'researcher',+ prompt: `Analyze existing authentication in ${authDir}.+ Find: login flow, logout flow, token handling.+ Return: structured findings only. NO implementation suggestions.`,+ toolFilter: ['read', 'search', 'glob'], // Can't write+});++await coordinator.synthesize(research.results);++const implement = await coordinator.spawn({+ role: 'implementer',+ prompt: `Implement token refresh handler using the JWT strategy+ from [Phase 2 findings].+ Constraints: Use existing AuthService patterns, add tests.`,+ toolFilter: ['read', 'search', 'edit', 'test'], // Can write+});+```++### Fork Pattern (Single-Level Only)++```typescript+// Parent spawns children for parallel work+const forks = await Promise.all([+ coordinator.fork({+ prompt: 'Implement login handler',+ inheritContext: true, // Full parent history+ }),+ coordinator.fork({+ prompt: 'Implement logout handler',+ inheritContext: true,+ }),+]);++// CRITICAL: Children must not fork recursively+// If allowed, context cost multiplies: parent + child1 + child2 + ...+```++### Swarm Pattern (Flat Roster)++```typescript+// Swarm: persistent team with shared task list+const swarm = new Swarm([+ { id: 'researcher', specialty: 'research' },+ { id: 'implementer', specialty: 'implementation' },+ { id: 'reviewer', specialty: 'verification' },+]);++// Agents pick tasks from shared queue+// Results posted back to shared state+await swarm.dispatch({+ taskId: 'feat-001',+ pickedBy: 'implementer',+});+```++## Gotchas++1. **Fork children must not fork** — Recursive guard preserves single-level invariant. Keep fork tool in child's pool (for prompt cache sharing) but block at call time.+2. **Coordinator workers start with zero context** — Only explicit prompt is passed. Don't assume child sees parent's accumulated research.+3. **Swarm teammates cannot spawn other teammates** — Roster is flat to prevent uncontrolled growth.+4. **Write self-contained prompts** — "Based on your findings" is an anti-pattern. Coordinator must digest first.+5. **Filter each worker's tool set** — Researcher doesn't need write; implementer doesn't need broad search.++## Related Patterns++- [Context Engineering](context-engineering-pattern.md) — Isolation patterns for delegation+- [Lifecycle & Bootstrap](lifecycle-bootstrap-pattern.md) — How agents are spawned at init++## Template: Worker Prompt Structure++```markdown+# Self-Contained Worker Prompt++## Context (Copied from Coordinator Synthesis)++**Task**: Implement token refresh handler+**Background**: Research identified JWT-based auth with 24h access tokens.+**Decision**: Use refresh token rotation (new refresh token on each refresh).++## Your Role++You are an **implementer**. Your job is to write production code following the specs above.++## Constraints++- Use existing patterns from `${authServicePath}`+- Add tests for success and failure cases+- Do NOT modify login/logout handlers (separate task)++## Your Tools++- read, search, edit, test+- Shell: npm test, npm run check only++## Deliverable++Return:+1. Implementation diff (files changed)+2. Test results (pass/fail)+3. Any blockers or clarifications needed++**Do NOT return**: Research findings, architectural debates, alternative designs.+```++## Evidence++Multi-agent coordination patterns are observed in production systems where:+- Coordinator workers start with zero context inheritance+- Fork is restricted to single-level to control context explosion+- Swarm agents communicate through shared task lists, not direct prompts+- Results arrive asynchronously with fire-and-forget registrationdiff --git a/.opencode/skills/harness-creator/references/tool-registry-pattern.md b/.opencode/skills/harness-creator/references/tool-registry-pattern.mdnew file mode 100644index 0000000..8bd42c7--- /dev/null+++ b/.opencode/skills/harness-creator/references/tool-registry-pattern.md@@ -0,0 +1,200 @@+# Tool Registry and Safety Pattern++## Problem++Agents need tools (shell, file edit, search, etc.) to be productive. But unbounded tool access creates risks:++- Destructive operations (rm -rf, DROP TABLE, etc.)+- Race conditions from concurrent tool calls+- Silent policy violations from misconfigured permissions++The solution is a **fail-closed registry** with explicit concurrency classification and a multi-source permission pipeline.++## Golden Rules++### Default to Fail-Closed++Tools are **non-concurrent** and **non-read-only** unless explicitly marked safe. This prevents:+- Accidental parallel execution of state-mutating operations+- Silent data corruption from concurrent writes++### Concurrency is Per-Call, Not Per-Tool++The same tool can be safe for some inputs and unsafe for others:++```+✓ Safe (can run in parallel):+ - cat file1.txt+ - grep "pattern" src/+ - ls -la++✗ Unsafe (must run serially):+ - rm -rf build/+ - npm install (network, filesystem mutation)+ - sed -i 's/old/new/g' *.ts+```++The runtime partitions a batch of tool calls into consecutive groups: safe calls run in parallel; any unsafe call starts a serial segment.++### Permission Pipeline has Side Effects++The permission evaluator is **stateful** — it:+- Tracks denials (for audit and rate limiting)+- Transforms modes (e.g., auto → ask after denial)+- Updates session state as a side effect++**Strict priority order:**+```+Policy (org-wide) → User settings → Project rules → Local overrides → Session grants+```++## When To Use++- Your agent runtime needs tool registration+- You need concurrency control for parallel tool calls+- You need permission gating (auto-approve, ask-first, deny)+- You need to track tool usage for audit++## Tradeoffs++| Decision | Benefit | Cost |+|---|---|---|+| Fail-closed defaults | New tools are safe out of the box | Developers must actively opt into concurrency |+| Per-call classification | Fine-grained control over parallelism | Requires analyzing each call, not just tool registration |+| Multi-source permission layering | Flexible policy composition | Hard to debug when rules conflict |+| Stateful evaluator | Can adapt behavior based on history | Not a pure function — harder to test |++## Implementation Patterns++### Tool Registration++```typescript+// Example: Tool registry entry+interface ToolDefinition {+ name: string;+ description: string;+ handler: (args: any) => Promise<any>;++ // Safety classification+ isReadOnly: boolean; // Default: false+ isConcurrentSafe: boolean; // Default: false++ // Optional custom permission logic+ permissionCheck?: (args: any, context: ToolContext) => PermissionResult;+}++// Register tools+registry.register('read_file', {+ name: 'read_file',+ description: 'Read contents of a file',+ handler: readFile,+ isReadOnly: true,+ isConcurrentSafe: true, // Safe to read multiple files in parallel+});++registry.register('write_file', {+ name: 'write_file',+ description: 'Write or overwrite a file',+ handler: writeFile,+ isReadOnly: false,+ isConcurrentSafe: false, // Must run serially to prevent race conditions+});+```++### Permission Pipeline++```typescript+// Permission evaluation order+async function evaluatePermission(+ toolCall: ToolCall,+ context: PermissionContext+): Promise<PermissionResult> {++ // 1. Policy rules (highest priority, org-wide)+ const policyResult = await policyEngine.check(toolCall, context);+ if (policyResult !== 'defer') return policyResult;++ // 2. User settings+ const userResult = await userSettings.check(toolCall, context);+ if (userResult !== 'defer') return userResult;++ // 3. Project rules+ const projectResult = await projectRules.check(toolCall, context);+ if (projectResult !== 'defer') return projectResult;++ // 4. Local overrides+ const localResult = await localOverrides.check(toolCall, context);+ if (localResult !== 'defer') return localResult;++ // 5. Session grants (lowest priority)+ return sessionGrants.check(toolCall, context);+}+```++### Bypass-Immune Rules++Certain paths or operations should never be auto-approved:++```yaml+# Protected paths (never auto-approve)+protected_paths:+ - /etc/**+ - /usr/**+ - node_modules/**+ - .git/**++# Protected commands (always ask)+protected_commands:+ - "rm -rf*"+ - "DROP TABLE*"+ - "DELETE FROM*"+ - "mkfs*"+```++## Gotchas++1. **Most async work skips "pending" state** — work units register directly as "running"+2. **Permission evaluation has side effects** — don't cache results across calls+3. **Concurrency classification requires analyzing inputs**, not just tool name+4. **The default permission for tools is "allow"** — tools without custom logic delegate to rule-based system+5. **Eviction requires notification** — terminal work units only GC-eligible after parent notified++## Related Patterns++- [Lifecycle & Bootstrap](lifecycle-bootstrap-pattern.md) — How tools are registered at init+- [Hook Lifecycle](hook-lifecycle-pattern.md) — Pre/post tool execution hooks++## Template: Tool Safety Checklist++Before enabling a new tool:++```markdown+## Tool Safety Review++**Tool name**: [e.g., execute_shell]++### Classification+- [ ] Determined if read-only (true / false / depends on args)+- [ ] Determined if concurrent-safe (true / false / depends on args)+- [ ] Documented unsafe input patterns++### Permission Requirements+- [ ] Default mode set to "ask" or "deny"+- [ ] Bypass-immune paths/commands defined+- [ ] Custom permission logic implemented (if needed)+- [ ] Audit logging enabled++### Testing+- [ ] Tested with safe inputs (should auto-approve)+- [ ] Tested with unsafe inputs (should ask/deny)+- [ ] Tested concurrent execution (should serialize if unsafe)+- [ ] Tested error handling (failures logged, state consistent)+```++## Evidence++Tool registry and safety patterns are observed in production agent runtimes including:+- Claude Code's tool registry with explicit concurrency flags+- Multi-source permission evaluation (settings → project → session)+- Protected path/command lists that bypass auto-approve modes+- Per-call concurrency classification that partitions tool batchesdiff --git a/.opencode/skills/harness-creator/templates/agents.md b/.opencode/skills/harness-creator/templates/agents.mdnew file mode 100644index 0000000..bae9a82--- /dev/null+++ b/.opencode/skills/harness-creator/templates/agents.md@@ -0,0 +1,68 @@+# AGENTS.md++[One-sentence project purpose]++## Startup Workflow++Before writing code:++1. **Confirm working directory** with `pwd`+2. **Read this file** completely+3. **Read [ARCHITECTURE.md / CLAUDE.md]** for system map and hard rules+4. **Run `./init.sh`** to verify environment is healthy+5. **Read `feature_list.json`** to see current feature state+6. **Review recent commits** with `git log --oneline -5`++If baseline verification is failing, repair that first before adding new scope.++## Working Rules++- **One feature at a time**: Pick exactly one unfinished feature from `feature_list.json`+- **Verification required**: Don't claim done without running verification commands+- **Update artifacts**: Before ending session, update `progress.md` and `feature_list.json`+- **Stay in scope**: Don't modify files unrelated to the current feature+- **Leave clean state**: Next session must be able to run `./init.sh` immediately++## Required Artifacts++- `feature_list.json` — Feature state tracker (source of truth)+- `progress.md` — Session continuity log+- `init.sh` — Standard startup and verification path+- `session-handoff.md` — Optional, for larger sessions++## Definition of Done++A feature is done only when ALL of the following are true:++- [ ] Target behavior is implemented+- [ ] Required verification actually ran (tests / lint / type-check)+- [ ] Evidence recorded in `feature_list.json` or `progress.md`+- [ ] Repository remains restartable from standard startup path++## End of Session++Before ending a session:++1. Update `progress.md` with current state+2. Update `feature_list.json` with new feature status+3. Record any unresolved risks or blockers+4. Commit with descriptive message once work is in safe state+5. Leave repo clean enough for next session to run `./init.sh` immediately++## Verification Commands++```bash+# Full verification (recommended)+./init.sh++# Individual checks+npm install && npm run check && npm test+```++## Escalation++If you encounter:+- **Architecture decisions**: Consult [ARCHITECTURE.md] or ask user+- **Unclear requirements**: Check [docs/PRODUCT.md] or ask user+- **Repeated test failures**: Update progress, flag for human review+- **Scope ambiguity**: Re-read `feature_list.json` for definition of donediff --git a/.opencode/skills/harness-creator/templates/feature-list.json b/.opencode/skills/harness-creator/templates/feature-list.jsonnew file mode 100644index 0000000..d0f369f--- /dev/null+++ b/.opencode/skills/harness-creator/templates/feature-list.json@@ -0,0 +1,44 @@+{+ "features": [+ {+ "id": "feat-001",+ "name": "Document Import",+ "description": "Allow users to import PDF and TXT documents from local filesystem",+ "dependencies": [],+ "status": "not-started",+ "evidence": ""+ },+ {+ "id": "feat-002",+ "name": "Document Chunking",+ "description": "Split imported documents into ~500 character chunks with metadata (word count, line count, paragraph count)",+ "dependencies": ["feat-001"],+ "status": "not-started",+ "evidence": ""+ },+ {+ "id": "feat-003",+ "name": "Indexing Status UI",+ "description": "Display indexing progress in status bar: total docs, indexed docs, last sync time",+ "dependencies": ["feat-002"],+ "status": "not-started",+ "evidence": ""+ },+ {+ "id": "feat-004",+ "name": "Grounded Q&A",+ "description": "Run AI-powered Q&A over indexed content with citations to source documents",+ "dependencies": ["feat-002"],+ "status": "not-started",+ "evidence": ""+ },+ {+ "id": "feat-005",+ "name": "Conversation History",+ "description": "Display Q&A conversation in chat-style interface with copy/export functionality",+ "dependencies": ["feat-004"],+ "status": "not-started",+ "evidence": ""+ }+ ]+}diff --git a/.opencode/skills/harness-creator/templates/feature-list.schema.json b/.opencode/skills/harness-creator/templates/feature-list.schema.jsonnew file mode 100644index 0000000..45b8521--- /dev/null+++ b/.opencode/skills/harness-creator/templates/feature-list.schema.json@@ -0,0 +1,47 @@+{+ "$schema": "http://json-schema.org/draft-07/schema#",+ "title": "Feature List",+ "description": "Track feature implementation state for agent-driven development",+ "type": "object",+ "properties": {+ "features": {+ "type": "array",+ "items": {+ "type": "object",+ "properties": {+ "id": {+ "type": "string",+ "description": "Unique feature identifier (e.g., feat-001)",+ "pattern": "^feat-\\d+$"+ },+ "name": {+ "type": "string",+ "description": "Short feature name"+ },+ "description": {+ "type": "string",+ "description": "What this feature does"+ },+ "dependencies": {+ "type": "array",+ "items": {+ "type": "string"+ },+ "description": "Feature IDs that must be done before this one"+ },+ "status": {+ "type": "string",+ "enum": ["not-started", "in-progress", "blocked", "done"],+ "description": "Current implementation state"+ },+ "evidence": {+ "type": "string",+ "description": "Verification evidence when status is 'done'"+ }+ },+ "required": ["id", "name", "description", "status"]+ }+ }+ },+ "required": ["features"]+}diff --git a/.opencode/skills/harness-creator/templates/init.sh b/.opencode/skills/harness-creator/templates/init.shnew file mode 100644index 0000000..e866eae--- /dev/null+++ b/.opencode/skills/harness-creator/templates/init.sh@@ -0,0 +1,17 @@+#!/bin/bash+set -e++echo "=== Harness Initialization ==="++npm install+npm run check+npm test+npm run build++echo "=== Verification Complete ==="+echo ""+echo "Next steps:"+echo "1. Read feature_list.json to see current feature state"+echo "2. Pick ONE unfinished feature to work on"+echo "3. Implement only that feature"+echo "4. Re-run verification before claiming done"diff --git a/.opencode/skills/harness-creator/templates/progress.md b/.opencode/skills/harness-creator/templates/progress.mdnew file mode 100644index 0000000..3710d5c--- /dev/null+++ b/.opencode/skills/harness-creator/templates/progress.md@@ -0,0 +1,51 @@+# Session Progress Log++## Current State++**Last Updated:** YYYY-MM-DD HH:MM+**Session ID:** [optional]+**Active Feature:** [feat-XXX - Feature Name]++## Status++### What's Done++- [x] [Completed item 1]+- [x] [Completed item 2]++### What's In Progress++- [ ] [Current work item]+ - Details: [specific task]+ - Blockers: [if any]++### What's Next++1. [Next action item]+2. [Following action item]++## Blockers / Risks++- [ ] [Blocker 1]: [description, impact]+- [ ] [Risk 1]: [description, mitigation]++## Decisions Made++- **[Decision 1]**: [description]+ - Context: [why this decision was made]+ - Alternatives considered: [what else was discussed]++## Files Modified This Session++- `path/to/file1.ts` - [brief description of change]+- `path/to/file2.ts` - [brief description of change]++## Evidence of Completion++- [ ] Tests pass: `[command and output]`+- [ ] Type check clean: `[command and output]`+- [ ] Manual verification: `[what was tested]`++## Notes for Next Session++[Free-form notes that will help the next session pick up context]diff --git a/.opencode/skills/skill-creator/SKILL.md b/.opencode/skills/skill-creator/SKILL.mdnew file mode 100644index 0000000..65b3a40--- /dev/null+++ b/.opencode/skills/skill-creator/SKILL.md@@ -0,0 +1,485 @@+---+name: skill-creator+description: Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.+---++# Skill Creator++A skill for creating new skills and iteratively improving them.++At a high level, the process of creating a skill goes like this:++- Decide what you want the skill to do and roughly how it should do it+- Write a draft of the skill+- Create a few test prompts and run claude-with-access-to-the-skill on them+- Help the user evaluate the results both qualitatively and quantitatively+ - While the runs happen in the background, draft some quantitative evals if there aren't any (if there are some, you can either use as is or modify if you feel something needs to change about them). Then explain them to the user (or if they already existed, explain the ones that already exist)+ - Use the `eval-viewer/generate_review.py` script to show the user the results for them to look at, and also let them look at the quantitative metrics+- Rewrite the skill based on feedback from the user's evaluation of the results (and also if there are any glaring flaws that become apparent from the quantitative benchmarks)+- Repeat until you're satisfied+- Expand the test set and try again at larger scale++Your job when using this skill is to figure out where the user is in this process and then jump in and help them progress through these stages. So for instance, maybe they're like "I want to make a skill for X". You can help narrow down what they mean, write a draft, write the test cases, figure out how they want to evaluate, run all the prompts, and repeat.++On the other hand, maybe they already have a draft of the skill. In this case you can go straight to the eval/iterate part of the loop.++Of course, you should always be flexible and if the user is like "I don't need to run a bunch of evaluations, just vibe with me", you can do that instead.++Then after the skill is done (but again, the order is flexible), you can also run the skill description improver, which we have a whole separate script for, to optimize the triggering of the skill.++Cool? Cool.++## Communicating with the user++The skill creator is liable to be used by people across a wide range of familiarity with coding jargon. If you haven't heard (and how could you, it's only very recently that it started), there's a trend now where the power of Claude is inspiring plumbers to open up their terminals, parents and grandparents to google "how to install npm". On the other hand, the bulk of users are probably fairly computer-literate.++So please pay attention to context cues to understand how to phrase your communication! In the default case, just to give you some idea:++- "evaluation" and "benchmark" are borderline, but OK+- for "JSON" and "assertion" you want to see serious cues from the user that they know what those things are before using them without explaining them++It's OK to briefly explain terms if you're in doubt, and feel free to clarify terms with a short definition if you're unsure if the user will get it.++---++## Creating a skill++### Capture Intent++Start by understanding the user's intent. The current conversation might already contain a workflow the user wants to capture (e.g., they say "turn this into a skill"). If so, extract answers from the conversation history first — the tools used, the sequence of steps, corrections the user made, input/output formats observed. The user may need to fill the gaps, and should confirm before proceeding to the next step.++1. What should this skill enable Claude to do?+2. When should this skill trigger? (what user phrases/contexts)+3. What's the expected output format?+4. Should we set up test cases to verify the skill works? Skills with objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps) benefit from test cases. Skills with subjective outputs (writing style, art) often don't need them. Suggest the appropriate default based on the skill type, but let the user decide.++### Interview and Research++Proactively ask questions about edge cases, input/output formats, example files, success criteria, and dependencies. Wait to write test prompts until you've got this part ironed out.++Check available MCPs - if useful for research (searching docs, finding similar skills, looking up best practices), research in parallel via subagents if available, otherwise inline. Come prepared with context to reduce burden on the user.++### Write the SKILL.md++Based on the user interview, fill in these components:++- **name**: Skill identifier+- **description**: When to trigger, what it does. This is the primary triggering mechanism - include both what the skill does AND specific contexts for when to use it. All "when to use" info goes here, not in the body. Note: currently Claude has a tendency to "undertrigger" skills -- to not use them when they'd be useful. To combat this, please make the skill descriptions a little bit "pushy". So for instance, instead of "How to build a simple fast dashboard to display internal Anthropic data.", you might write "How to build a simple fast dashboard to display internal Anthropic data. Make sure to use this skill whenever the user mentions dashboards, data visualization, internal metrics, or wants to display any kind of company data, even if they don't explicitly ask for a 'dashboard.'"+- **compatibility**: Required tools, dependencies (optional, rarely needed)+- **the rest of the skill :)**++### Skill Writing Guide++#### Anatomy of a Skill++```+skill-name/+├── SKILL.md (required)+│ ├── YAML frontmatter (name, description required)+│ └── Markdown instructions+└── Bundled Resources (optional)+ ├── scripts/ - Executable code for deterministic/repetitive tasks+ ├── references/ - Docs loaded into context as needed+ └── assets/ - Files used in output (templates, icons, fonts)+```++#### Progressive Disclosure++Skills use a three-level loading system:+1. **Metadata** (name + description) - Always in context (~100 words)+2. **SKILL.md body** - In context whenever skill triggers (<500 lines ideal)+3. **Bundled resources** - As needed (unlimited, scripts can execute without loading)++These word counts are approximate and you can feel free to go longer if needed.++**Key patterns:**+- Keep SKILL.md under 500 lines; if you're approaching this limit, add an additional layer of hierarchy along with clear pointers about where the model using the skill should go next to follow up.+- Reference files clearly from SKILL.md with guidance on when to read them+- For large reference files (>300 lines), include a table of contents++**Domain organization**: When a skill supports multiple domains/frameworks, organize by variant:+```+cloud-deploy/+├── SKILL.md (workflow + selection)+└── references/+ ├── aws.md+ ├── gcp.md+ └── azure.md+```+Claude reads only the relevant reference file.++#### Principle of Lack of Surprise++This goes without saying, but skills must not contain malware, exploit code, or any content that could compromise system security. A skill's contents should not surprise the user in their intent if described. Don't go along with requests to create misleading skills or skills designed to facilitate unauthorized access, data exfiltration, or other malicious activities. Things like a "roleplay as an XYZ" are OK though.++#### Writing Patterns++Prefer using the imperative form in instructions.++**Defining output formats** - You can do it like this:+```markdown+## Report structure+ALWAYS use this exact template:+# [Title]+## Executive summary+## Key findings+## Recommendations+```++**Examples pattern** - It's useful to include examples. You can format them like this (but if "Input" and "Output" are in the examples you might want to deviate a little):+```markdown+## Commit message format+**Example 1:**+Input: Added user authentication with JWT tokens+Output: feat(auth): implement JWT-based authentication+```++### Writing Style++Try to explain to the model why things are important in lieu of heavy-handed musty MUSTs. Use theory of mind and try to make the skill general and not super-narrow to specific examples. Start by writing a draft and then look at it with fresh eyes and improve it.++### Test Cases++After writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Share them with the user: [you don't have to use this exact language] "Here are a few test cases I'd like to try. Do these look right, or do you want to add more?" Then run them.++Save test cases to `evals/evals.json`. Don't write assertions yet — just the prompts. You'll draft assertions in the next step while the runs are in progress.++```json+{+ "skill_name": "example-skill",+ "evals": [+ {+ "id": 1,+ "prompt": "User's task prompt",+ "expected_output": "Description of expected result",+ "files": []+ }+ ]+}+```++See `references/schemas.md` for the full schema (including the `assertions` field, which you'll add later).++## Running and evaluating test cases++This section is one continuous sequence — don't stop partway through. Do NOT use `/skill-test` or any other testing skill.++Put results in `<skill-name>-workspace/` as a sibling to the skill directory. Within the workspace, organize results by iteration (`iteration-1/`, `iteration-2/`, etc.) and within that, each test case gets a directory (`eval-0/`, `eval-1/`, etc.). Don't create all of this upfront — just create directories as you go.++### Step 1: Spawn all runs (with-skill AND baseline) in the same turn++For each test case, spawn two subagents in the same turn — one with the skill, one without. This is important: don't spawn the with-skill runs first and then come back for baselines later. Launch everything at once so it all finishes around the same time.++**With-skill run:**++```+Execute this task:+- Skill path: <path-to-skill>+- Task: <eval prompt>+- Input files: <eval files if any, or "none">+- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/+- Outputs to save: <what the user cares about — e.g., "the .docx file", "the final CSV">+```++**Baseline run** (same prompt, but the baseline depends on context):+- **Creating a new skill**: no skill at all. Same prompt, no skill path, save to `without_skill/outputs/`.+- **Improving an existing skill**: the old version. Before editing, snapshot the skill (`cp -r <skill-path> <workspace>/skill-snapshot/`), then point the baseline subagent at the snapshot. Save to `old_skill/outputs/`.++Write an `eval_metadata.json` for each test case (assertions can be empty for now). Give each eval a descriptive name based on what it's testing — not just "eval-0". Use this name for the directory too. If this iteration uses new or modified eval prompts, create these files for each new eval directory — don't assume they carry over from previous iterations.++```json+{+ "eval_id": 0,+ "eval_name": "descriptive-name-here",+ "prompt": "The user's task prompt",+ "assertions": []+}+```++### Step 2: While runs are in progress, draft assertions++Don't just wait for the runs to finish — you can use this time productively. Draft quantitative assertions for each test case and explain them to the user. If assertions already exist in `evals/evals.json`, review them and explain what they check.++Good assertions are objectively verifiable and have descriptive names — they should read clearly in the benchmark viewer so someone glancing at the results immediately understands what each one checks. Subjective skills (writing style, design quality) are better evaluated qualitatively — don't force assertions onto things that need human judgment.++Update the `eval_metadata.json` files and `evals/evals.json` with the assertions once drafted. Also explain to the user what they'll see in the viewer — both the qualitative outputs and the quantitative benchmark.++### Step 3: As runs complete, capture timing data++When each subagent task completes, you receive a notification containing `total_tokens` and `duration_ms`. Save this data immediately to `timing.json` in the run directory:++```json+{+ "total_tokens": 84852,+ "duration_ms": 23332,+ "total_duration_seconds": 23.3+}+```++This is the only opportunity to capture this data — it comes through the task notification and isn't persisted elsewhere. Process each notification as it arrives rather than trying to batch them.++### Step 4: Grade, aggregate, and launch the viewer++Once all runs are done:++1. **Grade each run** — spawn a grader subagent (or grade inline) that reads `agents/grader.md` and evaluates each assertion against the outputs. Save results to `grading.json` in each run directory. The grading.json expectations array must use the fields `text`, `passed`, and `evidence` (not `name`/`met`/`details` or other variants) — the viewer depends on these exact field names. For assertions that can be checked programmatically, write and run a script rather than eyeballing it — scripts are faster, more reliable, and can be reused across iterations.++2. **Aggregate into benchmark** — run the aggregation script from the skill-creator directory:+ ```bash+ python -m scripts.aggregate_benchmark <workspace>/iteration-N --skill-name <name>+ ```+ This produces `benchmark.json` and `benchmark.md` with pass_rate, time, and tokens for each configuration, with mean ± stddev and the delta. If generating benchmark.json manually, see `references/schemas.md` for the exact schema the viewer expects.+Put each with_skill version before its baseline counterpart.++3. **Do an analyst pass** — read the benchmark data and surface patterns the aggregate stats might hide. See `agents/analyzer.md` (the "Analyzing Benchmark Results" section) for what to look for — things like assertions that always pass regardless of skill (non-discriminating), high-variance evals (possibly flaky), and time/token tradeoffs.++4. **Launch the viewer** with both qualitative outputs and quantitative data:+ ```bash+ nohup python <skill-creator-path>/eval-viewer/generate_review.py \+ <workspace>/iteration-N \+ --skill-name "my-skill" \+ --benchmark <workspace>/iteration-N/benchmark.json \+ > /dev/null 2>&1 &+ VIEWER_PID=$!+ ```+ For iteration 2+, also pass `--previous-workspace <workspace>/iteration-<N-1>`.++ **Cowork / headless environments:** If `webbrowser.open()` is not available or the environment has no display, use `--static <output_path>` to write a standalone HTML file instead of starting a server. Feedback will be downloaded as a `feedback.json` file when the user clicks "Submit All Reviews". After download, copy `feedback.json` into the workspace directory for the next iteration to pick up.++Note: please use generate_review.py to create the viewer; there's no need to write custom HTML.++5. **Tell the user** something like: "I've opened the results in your browser. There are two tabs — 'Outputs' lets you click through each test case and leave feedback, 'Benchmark' shows the quantitative comparison. When you're done, come back here and let me know."++### What the user sees in the viewer++The "Outputs" tab shows one test case at a time:+- **Prompt**: the task that was given+- **Output**: the files the skill produced, rendered inline where possible+- **Previous Output** (iteration 2+): collapsed section showing last iteration's output+- **Formal Grades** (if grading was run): collapsed section showing assertion pass/fail+- **Feedback**: a textbox that auto-saves as they type+- **Previous Feedback** (iteration 2+): their comments from last time, shown below the textbox++The "Benchmark" tab shows the stats summary: pass rates, timing, and token usage for each configuration, with per-eval breakdowns and analyst observations.++Navigation is via prev/next buttons or arrow keys. When done, they click "Submit All Reviews" which saves all feedback to `feedback.json`.++### Step 5: Read the feedback++When the user tells you they're done, read `feedback.json`:++```json+{+ "reviews": [+ {"run_id": "eval-0-with_skill", "feedback": "the chart is missing axis labels", "timestamp": "..."},+ {"run_id": "eval-1-with_skill", "feedback": "", "timestamp": "..."},+ {"run_id": "eval-2-with_skill", "feedback": "perfect, love this", "timestamp": "..."}+ ],+ "status": "complete"+}+```++Empty feedback means the user thought it was fine. Focus your improvements on the test cases where the user had specific complaints.++Kill the viewer server when you're done with it:++```bash+kill $VIEWER_PID 2>/dev/null+```++---++## Improving the skill++This is the heart of the loop. You've run the test cases, the user has reviewed the results, and now you need to make the skill better based on their feedback.++### How to think about improvements++1. **Generalize from the feedback.** The big picture thing that's happening here is that we're trying to create skills that can be used a million times (maybe literally, maybe even more who knows) across many different prompts. Here you and the user are iterating on only a few examples over and over again because it helps move faster. The user knows these examples in and out and it's quick for them to assess new outputs. But if the skill you and the user are codeveloping works only for those examples, it's useless. Rather than put in fiddly overfitty changes, or oppressively constrictive MUSTs, if there's some stubborn issue, you might try branching out and using different metaphors, or recommending different patterns of working. It's relatively cheap to try and maybe you'll land on something great.++2. **Keep the prompt lean.** Remove things that aren't pulling their weight. Make sure to read the transcripts, not just the final outputs — if it looks like the skill is making the model waste a bunch of time doing things that are unproductive, you can try getting rid of the parts of the skill that are making it do that and seeing what happens.++3. **Explain the why.** Try hard to explain the **why** behind everything you're asking the model to do. Today's LLMs are *smart*. They have good theory of mind and when given a good harness can go beyond rote instructions and really make things happen. Even if the feedback from the user is terse or frustrated, try to actually understand the task and why the user is writing what they wrote, and what they actually wrote, and then transmit this understanding into the instructions. If you find yourself writing ALWAYS or NEVER in all caps, or using super rigid structures, that's a yellow flag — if possible, reframe and explain the reasoning so that the model understands why the thing you're asking for is important. That's a more humane, powerful, and effective approach.++4. **Look for repeated work across test cases.** Read the transcripts from the test runs and notice if the subagents all independently wrote similar helper scripts or took the same multi-step approach to something. If all 3 test cases resulted in the subagent writing a `create_docx.py` or a `build_chart.py`, that's a strong signal the skill should bundle that script. Write it once, put it in `scripts/`, and tell the skill to use it. This saves every future invocation from reinventing the wheel.++This task is pretty important (we are trying to create billions a year in economic value here!) and your thinking time is not the blocker; take your time and really mull things over. I'd suggest writing a draft revision and then looking at it anew and making improvements. Really do your best to get into the head of the user and understand what they want and need.++### The iteration loop++After improving the skill:++1. Apply your improvements to the skill+2. Rerun all test cases into a new `iteration-<N+1>/` directory, including baseline runs. If you're creating a new skill, the baseline is always `without_skill` (no skill) — that stays the same across iterations. If you're improving an existing skill, use your judgment on what makes sense as the baseline: the original version the user came in with, or the previous iteration.+3. Launch the reviewer with `--previous-workspace` pointing at the previous iteration+4. Wait for the user to review and tell you they're done+5. Read the new feedback, improve again, repeat++Keep going until:+- The user says they're happy+- The feedback is all empty (everything looks good)+- You're not making meaningful progress++---++## Advanced: Blind comparison++For situations where you want a more rigorous comparison between two versions of a skill (e.g., the user asks "is the new version actually better?"), there's a blind comparison system. Read `agents/comparator.md` and `agents/analyzer.md` for the details. The basic idea is: give two outputs to an independent agent without telling it which is which, and let it judge quality. Then analyze why the winner won.++This is optional, requires subagents, and most users won't need it. The human review loop is usually sufficient.++---++## Description Optimization++The description field in SKILL.md frontmatter is the primary mechanism that determines whether Claude invokes a skill. After creating or improving a skill, offer to optimize the description for better triggering accuracy.++### Step 1: Generate trigger eval queries++Create 20 eval queries — a mix of should-trigger and should-not-trigger. Save as JSON:++```json+[+ {"query": "the user prompt", "should_trigger": true},+ {"query": "another prompt", "should_trigger": false}+]+```++The queries must be realistic and something a Claude Code or Claude.ai user would actually type. Not abstract requests, but requests that are concrete and specific and have a good amount of detail. For instance, file paths, personal context about the user's job or situation, column names and values, company names, URLs. A little bit of backstory. Some might be in lowercase or contain abbreviations or typos or casual speech. Use a mix of different lengths, and focus on edge cases rather than making them clear-cut (the user will get a chance to sign off on them).++Bad: `"Format this data"`, `"Extract text from PDF"`, `"Create a chart"`++Good: `"ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add a column that shows the profit margin as a percentage. The revenue is in column C and costs are in column D i think"`++For the **should-trigger** queries (8-10), think about coverage. You want different phrasings of the same intent — some formal, some casual. Include cases where the user doesn't explicitly name the skill or file type but clearly needs it. Throw in some uncommon use cases and cases where this skill competes with another but should win.++For the **should-not-trigger** queries (8-10), the most valuable ones are the near-misses — queries that share keywords or concepts with the skill but actually need something different. Think adjacent domains, ambiguous phrasing where a naive keyword match would trigger but shouldn't, and cases where the query touches on something the skill does but in a context where another tool is more appropriate.++The key thing to avoid: don't make should-not-trigger queries obviously irrelevant. "Write a fibonacci function" as a negative test for a PDF skill is too easy — it doesn't test anything. The negative cases should be genuinely tricky.++### Step 2: Review with user++Present the eval set to the user for review using the HTML template:++1. Read the template from `assets/eval_review.html`+2. Replace the placeholders:+ - `__EVAL_DATA_PLACEHOLDER__` → the JSON array of eval items (no quotes around it — it's a JS variable assignment)+ - `__SKILL_NAME_PLACEHOLDER__` → the skill's name+ - `__SKILL_DESCRIPTION_PLACEHOLDER__` → the skill's current description+3. Write to a temp file (e.g., `/tmp/eval_review_<skill-name>.html`) and open it: `open /tmp/eval_review_<skill-name>.html`+4. The user can edit queries, toggle should-trigger, add/remove entries, then click "Export Eval Set"+5. The file downloads to `~/Downloads/eval_set.json` — check the Downloads folder for the most recent version in case there are multiple (e.g., `eval_set (1).json`)++This step matters — bad eval queries lead to bad descriptions.++### Step 3: Run the optimization loop++Tell the user: "This will take some time — I'll run the optimization loop in the background and check on it periodically."++Save the eval set to the workspace, then run in the background:++```bash+python -m scripts.run_loop \+ --eval-set <path-to-trigger-eval.json> \+ --skill-path <path-to-skill> \+ --model <model-id-powering-this-session> \+ --max-iterations 5 \+ --verbose+```++Use the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences.++While it runs, periodically tail the output to give the user updates on which iteration it's on and what the scores look like.++This handles the full optimization loop automatically. It splits the eval set into 60% train and 40% held-out test, evaluates the current description (running each query 3 times to get a reliable trigger rate), then calls Claude to propose improvements based on what failed. It re-evaluates each new description on both train and test, iterating up to 5 times. When it's done, it opens an HTML report in the browser showing the results per iteration and returns JSON with `best_description` — selected by test score rather than train score to avoid overfitting.++### How skill triggering works++Understanding the triggering mechanism helps design better eval queries. Skills appear in Claude's `available_skills` list with their name + description, and Claude decides whether to consult a skill based on that description. The important thing to know is that Claude only consults skills for tasks it can't easily handle on its own — simple, one-step queries like "read this PDF" may not trigger a skill even if the description matches perfectly, because Claude can handle them directly with basic tools. Complex, multi-step, or specialized queries reliably trigger skills when the description matches.++This means your eval queries should be substantive enough that Claude would actually benefit from consulting a skill. Simple queries like "read file X" are poor test cases — they won't trigger skills regardless of description quality.++### Step 4: Apply the result++Take `best_description` from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores.++---++### Package and Present (only if `present_files` tool is available)++Check whether you have access to the `present_files` tool. If you don't, skip this step. If you do, package the skill and present the .skill file to the user:++```bash+python -m scripts.package_skill <path/to/skill-folder>+```++After packaging, direct the user to the resulting `.skill` file path so they can install it.++---++## Claude.ai-specific instructions++In Claude.ai, the core workflow is the same (draft → test → review → improve → repeat), but because Claude.ai doesn't have subagents, some mechanics change. Here's what to adapt:++**Running test cases**: No subagents means no parallel execution. For each test case, read the skill's SKILL.md, then follow its instructions to accomplish the test prompt yourself. Do them one at a time. This is less rigorous than independent subagents (you wrote the skill and you're also running it, so you have full context), but it's a useful sanity check — and the human review step compensates. Skip the baseline runs — just use the skill to complete the task as requested.++**Reviewing results**: If you can't open a browser (e.g., Claude.ai's VM has no display, or you're on a remote server), skip the browser reviewer entirely. Instead, present results directly in the conversation. For each test case, show the prompt and the output. If the output is a file the user needs to see (like a .docx or .xlsx), save it to the filesystem and tell them where it is so they can download and inspect it. Ask for feedback inline: "How does this look? Anything you'd change?"++**Benchmarking**: Skip the quantitative benchmarking — it relies on baseline comparisons which aren't meaningful without subagents. Focus on qualitative feedback from the user.++**The iteration loop**: Same as before — improve the skill, rerun the test cases, ask for feedback — just without the browser reviewer in the middle. You can still organize results into iteration directories on the filesystem if you have one.++**Description optimization**: This section requires the `claude` CLI tool (specifically `claude -p`) which is only available in Claude Code. Skip it if you're on Claude.ai.++**Blind comparison**: Requires subagents. Skip it.++**Packaging**: The `package_skill.py` script works anywhere with Python and a filesystem. On Claude.ai, you can run it and the user can download the resulting `.skill` file.++**Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. In this case:+- **Preserve the original name.** Note the skill's directory name and `name` frontmatter field -- use them unchanged. E.g., if the installed skill is `research-helper`, output `research-helper.skill` (not `research-helper-v2`).+- **Copy to a writeable location before editing.** The installed skill path may be read-only. Copy to `/tmp/skill-name/`, edit there, and package from the copy.+- **If packaging manually, stage in `/tmp/` first**, then copy to the output directory -- direct writes may fail due to permissions.++---++## Cowork-Specific Instructions++If you're in Cowork, the main things to know are:++- You have subagents, so the main workflow (spawn test cases in parallel, run baselines, grade, etc.) all works. (However, if you run into severe problems with timeouts, it's OK to run the test prompts in series rather than parallel.)+- You don't have a browser or display, so when generating the eval viewer, use `--static <output_path>` to write a standalone HTML file instead of starting a server. Then proffer a link that the user can click to open the HTML in their browser.+- For whatever reason, the Cowork setup seems to disincline Claude from generating the eval viewer after running the tests, so just to reiterate: whether you're in Cowork or in Claude Code, after running tests, you should always generate the eval viewer for the human to look at examples before revising the skill yourself and trying to make corrections, using `generate_review.py` (not writing your own boutique html code). Sorry in advance but I'm gonna go all caps here: GENERATE THE EVAL VIEWER *BEFORE* evaluating inputs yourself. You want to get them in front of the human ASAP!+- Feedback works differently: since there's no running server, the viewer's "Submit All Reviews" button will download `feedback.json` as a file. You can then read it from there (you may have to request access first).+- Packaging works — `package_skill.py` just needs Python and a filesystem.+- Description optimization (`run_loop.py` / `run_eval.py`) should work in Cowork just fine since it uses `claude -p` via subprocess, not a browser, but please save it until you've fully finished making the skill and the user agrees it's in good shape.+- **Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. Follow the update guidance in the claude.ai section above.++---++## Reference files++The agents/ directory contains instructions for specialized subagents. Read them when you need to spawn the relevant subagent.++- `agents/grader.md` — How to evaluate assertions against outputs+- `agents/comparator.md` — How to do blind A/B comparison between two outputs+- `agents/analyzer.md` — How to analyze why one version beat another++The references/ directory has additional documentation:+- `references/schemas.md` — JSON structures for evals.json, grading.json, etc.++---++Repeating one more time the core loop here for emphasis:++- Figure out what the skill is about+- Draft or edit the skill+- Run claude-with-access-to-the-skill on test prompts+- With the user, evaluate the outputs:+ - Create benchmark.json and run `eval-viewer/generate_review.py` to help the user review them+ - Run quantitative evals+- Repeat until you and the user are satisfied+- Package the final skill and return it to the user.++Please add steps to your TodoList, if you have such a thing, to make sure you don't forget. If you're in Cowork, please specifically put "Create evals JSON and run `eval-viewer/generate_review.py` so human can review test cases" in your TodoList to make sure it happens.++Good luck!diff --git a/.opencode/skills/skill-creator/agents/analyzer.md b/.opencode/skills/skill-creator/agents/analyzer.mdnew file mode 100644index 0000000..14e41d6--- /dev/null+++ b/.opencode/skills/skill-creator/agents/analyzer.md@@ -0,0 +1,274 @@+# Post-hoc Analyzer Agent++Analyze blind comparison results to understand WHY the winner won and generate improvement suggestions.++## Role++After the blind comparator determines a winner, the Post-hoc Analyzer "unblids" the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved?++## Inputs++You receive these parameters in your prompt:++- **winner**: "A" or "B" (from blind comparison)+- **winner_skill_path**: Path to the skill that produced the winning output+- **winner_transcript_path**: Path to the execution transcript for the winner+- **loser_skill_path**: Path to the skill that produced the losing output+- **loser_transcript_path**: Path to the execution transcript for the loser+- **comparison_result_path**: Path to the blind comparator's output JSON+- **output_path**: Where to save the analysis results++## Process++### Step 1: Read Comparison Result++1. Read the blind comparator's output at comparison_result_path+2. Note the winning side (A or B), the reasoning, and any scores+3. Understand what the comparator valued in the winning output++### Step 2: Read Both Skills++1. Read the winner skill's SKILL.md and key referenced files+2. Read the loser skill's SKILL.md and key referenced files+3. Identify structural differences:+ - Instructions clarity and specificity+ - Script/tool usage patterns+ - Example coverage+ - Edge case handling++### Step 3: Read Both Transcripts++1. Read the winner's transcript+2. Read the loser's transcript+3. Compare execution patterns:+ - How closely did each follow their skill's instructions?+ - What tools were used differently?+ - Where did the loser diverge from optimal behavior?+ - Did either encounter errors or make recovery attempts?++### Step 4: Analyze Instruction Following++For each transcript, evaluate:+- Did the agent follow the skill's explicit instructions?+- Did the agent use the skill's provided tools/scripts?+- Were there missed opportunities to leverage skill content?+- Did the agent add unnecessary steps not in the skill?++Score instruction following 1-10 and note specific issues.++### Step 5: Identify Winner Strengths++Determine what made the winner better:+- Clearer instructions that led to better behavior?+- Better scripts/tools that produced better output?+- More comprehensive examples that guided edge cases?+- Better error handling guidance?++Be specific. Quote from skills/transcripts where relevant.++### Step 6: Identify Loser Weaknesses++Determine what held the loser back:+- Ambiguous instructions that led to suboptimal choices?+- Missing tools/scripts that forced workarounds?+- Gaps in edge case coverage?+- Poor error handling that caused failures?++### Step 7: Generate Improvement Suggestions++Based on the analysis, produce actionable suggestions for improving the loser skill:+- Specific instruction changes to make+- Tools/scripts to add or modify+- Examples to include+- Edge cases to address++Prioritize by impact. Focus on changes that would have changed the outcome.++### Step 8: Write Analysis Results++Save structured analysis to `{output_path}`.++## Output Format++Write a JSON file with this structure:++```json+{+ "comparison_summary": {+ "winner": "A",+ "winner_skill": "path/to/winner/skill",+ "loser_skill": "path/to/loser/skill",+ "comparator_reasoning": "Brief summary of why comparator chose winner"+ },+ "winner_strengths": [+ "Clear step-by-step instructions for handling multi-page documents",+ "Included validation script that caught formatting errors",+ "Explicit guidance on fallback behavior when OCR fails"+ ],+ "loser_weaknesses": [+ "Vague instruction 'process the document appropriately' led to inconsistent behavior",+ "No script for validation, agent had to improvise and made errors",+ "No guidance on OCR failure, agent gave up instead of trying alternatives"+ ],+ "instruction_following": {+ "winner": {+ "score": 9,+ "issues": [+ "Minor: skipped optional logging step"+ ]+ },+ "loser": {+ "score": 6,+ "issues": [+ "Did not use the skill's formatting template",+ "Invented own approach instead of following step 3",+ "Missed the 'always validate output' instruction"+ ]+ }+ },+ "improvement_suggestions": [+ {+ "priority": "high",+ "category": "instructions",+ "suggestion": "Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template",+ "expected_impact": "Would eliminate ambiguity that caused inconsistent behavior"+ },+ {+ "priority": "high",+ "category": "tools",+ "suggestion": "Add validate_output.py script similar to winner skill's validation approach",+ "expected_impact": "Would catch formatting errors before final output"+ },+ {+ "priority": "medium",+ "category": "error_handling",+ "suggestion": "Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'",+ "expected_impact": "Would prevent early failure on difficult documents"+ }+ ],+ "transcript_insights": {+ "winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output",+ "loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors"+ }+}+```++## Guidelines++- **Be specific**: Quote from skills and transcripts, don't just say "instructions were unclear"+- **Be actionable**: Suggestions should be concrete changes, not vague advice+- **Focus on skill improvements**: The goal is to improve the losing skill, not critique the agent+- **Prioritize by impact**: Which changes would most likely have changed the outcome?+- **Consider causation**: Did the skill weakness actually cause the worse output, or is it incidental?+- **Stay objective**: Analyze what happened, don't editorialize+- **Think about generalization**: Would this improvement help on other evals too?++## Categories for Suggestions++Use these categories to organize improvement suggestions:++| Category | Description |+|----------|-------------|+| `instructions` | Changes to the skill's prose instructions |+| `tools` | Scripts, templates, or utilities to add/modify |+| `examples` | Example inputs/outputs to include |+| `error_handling` | Guidance for handling failures |+| `structure` | Reorganization of skill content |+| `references` | External docs or resources to add |++## Priority Levels++- **high**: Would likely change the outcome of this comparison+- **medium**: Would improve quality but may not change win/loss+- **low**: Nice to have, marginal improvement++---++# Analyzing Benchmark Results++When analyzing benchmark results, the analyzer's purpose is to **surface patterns and anomalies** across multiple runs, not suggest skill improvements.++## Role++Review all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn't be visible from aggregate metrics alone.++## Inputs++You receive these parameters in your prompt:++- **benchmark_data_path**: Path to the in-progress benchmark.json with all run results+- **skill_path**: Path to the skill being benchmarked+- **output_path**: Where to save the notes (as JSON array of strings)++## Process++### Step 1: Read Benchmark Data++1. Read the benchmark.json containing all run results+2. Note the configurations tested (with_skill, without_skill)+3. Understand the run_summary aggregates already calculated++### Step 2: Analyze Per-Assertion Patterns++For each expectation across all runs:+- Does it **always pass** in both configurations? (may not differentiate skill value)+- Does it **always fail** in both configurations? (may be broken or beyond capability)+- Does it **always pass with skill but fail without**? (skill clearly adds value here)+- Does it **always fail with skill but pass without**? (skill may be hurting)+- Is it **highly variable**? (flaky expectation or non-deterministic behavior)++### Step 3: Analyze Cross-Eval Patterns++Look for patterns across evals:+- Are certain eval types consistently harder/easier?+- Do some evals show high variance while others are stable?+- Are there surprising results that contradict expectations?++### Step 4: Analyze Metrics Patterns++Look at time_seconds, tokens, tool_calls:+- Does the skill significantly increase execution time?+- Is there high variance in resource usage?+- Are there outlier runs that skew the aggregates?++### Step 5: Generate Notes++Write freeform observations as a list of strings. Each note should:+- State a specific observation+- Be grounded in the data (not speculation)+- Help the user understand something the aggregate metrics don't show++Examples:+- "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value"+- "Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure that may be flaky"+- "Without-skill runs consistently fail on table extraction expectations (0% pass rate)"+- "Skill adds 13s average execution time but improves pass rate by 50%"+- "Token usage is 80% higher with skill, primarily due to script output parsing"+- "All 3 without-skill runs for eval 1 produced empty output"++### Step 6: Write Notes++Save notes to `{output_path}` as a JSON array of strings:++```json+[+ "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value",+ "Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure",+ "Without-skill runs consistently fail on table extraction expectations",+ "Skill adds 13s average execution time but improves pass rate by 50%"+]+```++## Guidelines++**DO:**+- Report what you observe in the data+- Be specific about which evals, expectations, or runs you're referring to+- Note patterns that aggregate metrics would hide+- Provide context that helps interpret the numbers++**DO NOT:**+- Suggest improvements to the skill (that's for the improvement step, not benchmarking)+- Make subjective quality judgments ("the output was good/bad")+- Speculate about causes without evidence+- Repeat information already in the run_summary aggregatesdiff --git a/.opencode/skills/skill-creator/agents/comparator.md b/.opencode/skills/skill-creator/agents/comparator.mdnew file mode 100644index 0000000..80e00eb--- /dev/null+++ b/.opencode/skills/skill-creator/agents/comparator.md@@ -0,0 +1,202 @@+# Blind Comparator Agent++Compare two outputs WITHOUT knowing which skill produced them.++## Role++The Blind Comparator judges which output better accomplishes the eval task. You receive two outputs labeled A and B, but you do NOT know which skill produced which. This prevents bias toward a particular skill or approach.++Your judgment is based purely on output quality and task completion.++## Inputs++You receive these parameters in your prompt:++- **output_a_path**: Path to the first output file or directory+- **output_b_path**: Path to the second output file or directory+- **eval_prompt**: The original task/prompt that was executed+- **expectations**: List of expectations to check (optional - may be empty)++## Process++### Step 1: Read Both Outputs++1. Examine output A (file or directory)+2. Examine output B (file or directory)+3. Note the type, structure, and content of each+4. If outputs are directories, examine all relevant files inside++### Step 2: Understand the Task++1. Read the eval_prompt carefully+2. Identify what the task requires:+ - What should be produced?+ - What qualities matter (accuracy, completeness, format)?+ - What would distinguish a good output from a poor one?++### Step 3: Generate Evaluation Rubric++Based on the task, generate a rubric with two dimensions:++**Content Rubric** (what the output contains):+| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |+|-----------|----------|----------------|---------------|+| Correctness | Major errors | Minor errors | Fully correct |+| Completeness | Missing key elements | Mostly complete | All elements present |+| Accuracy | Significant inaccuracies | Minor inaccuracies | Accurate throughout |++**Structure Rubric** (how the output is organized):+| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |+|-----------|----------|----------------|---------------|+| Organization | Disorganized | Reasonably organized | Clear, logical structure |+| Formatting | Inconsistent/broken | Mostly consistent | Professional, polished |+| Usability | Difficult to use | Usable with effort | Easy to use |++Adapt criteria to the specific task. For example:+- PDF form → "Field alignment", "Text readability", "Data placement"+- Document → "Section structure", "Heading hierarchy", "Paragraph flow"+- Data output → "Schema correctness", "Data types", "Completeness"++### Step 4: Evaluate Each Output Against the Rubric++For each output (A and B):++1. **Score each criterion** on the rubric (1-5 scale)+2. **Calculate dimension totals**: Content score, Structure score+3. **Calculate overall score**: Average of dimension scores, scaled to 1-10++### Step 5: Check Assertions (if provided)++If expectations are provided:++1. Check each expectation against output A+2. Check each expectation against output B+3. Count pass rates for each output+4. Use expectation scores as secondary evidence (not the primary decision factor)++### Step 6: Determine the Winner++Compare A and B based on (in priority order):++1. **Primary**: Overall rubric score (content + structure)+2. **Secondary**: Assertion pass rates (if applicable)+3. **Tiebreaker**: If truly equal, declare a TIE++Be decisive - ties should be rare. One output is usually better, even if marginally.++### Step 7: Write Comparison Results++Save results to a JSON file at the path specified (or `comparison.json` if not specified).++## Output Format++Write a JSON file with this structure:++```json+{+ "winner": "A",+ "reasoning": "Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.",+ "rubric": {+ "A": {+ "content": {+ "correctness": 5,+ "completeness": 5,+ "accuracy": 4+ },+ "structure": {+ "organization": 4,+ "formatting": 5,+ "usability": 4+ },+ "content_score": 4.7,+ "structure_score": 4.3,+ "overall_score": 9.0+ },+ "B": {+ "content": {+ "correctness": 3,+ "completeness": 2,+ "accuracy": 3+ },+ "structure": {+ "organization": 3,+ "formatting": 2,+ "usability": 3+ },+ "content_score": 2.7,+ "structure_score": 2.7,+ "overall_score": 5.4+ }+ },+ "output_quality": {+ "A": {+ "score": 9,+ "strengths": ["Complete solution", "Well-formatted", "All fields present"],+ "weaknesses": ["Minor style inconsistency in header"]+ },+ "B": {+ "score": 5,+ "strengths": ["Readable output", "Correct basic structure"],+ "weaknesses": ["Missing date field", "Formatting inconsistencies", "Partial data extraction"]+ }+ },+ "expectation_results": {+ "A": {+ "passed": 4,+ "total": 5,+ "pass_rate": 0.80,+ "details": [+ {"text": "Output includes name", "passed": true},+ {"text": "Output includes date", "passed": true},+ {"text": "Format is PDF", "passed": true},+ {"text": "Contains signature", "passed": false},+ {"text": "Readable text", "passed": true}+ ]+ },+ "B": {+ "passed": 3,+ "total": 5,+ "pass_rate": 0.60,+ "details": [+ {"text": "Output includes name", "passed": true},+ {"text": "Output includes date", "passed": false},+ {"text": "Format is PDF", "passed": true},+ {"text": "Contains signature", "passed": false},+ {"text": "Readable text", "passed": true}+ ]+ }+ }+}+```++If no expectations were provided, omit the `expectation_results` field entirely.++## Field Descriptions++- **winner**: "A", "B", or "TIE"+- **reasoning**: Clear explanation of why the winner was chosen (or why it's a tie)+- **rubric**: Structured rubric evaluation for each output+ - **content**: Scores for content criteria (correctness, completeness, accuracy)+ - **structure**: Scores for structure criteria (organization, formatting, usability)+ - **content_score**: Average of content criteria (1-5)+ - **structure_score**: Average of structure criteria (1-5)+ - **overall_score**: Combined score scaled to 1-10+- **output_quality**: Summary quality assessment+ - **score**: 1-10 rating (should match rubric overall_score)+ - **strengths**: List of positive aspects+ - **weaknesses**: List of issues or shortcomings+- **expectation_results**: (Only if expectations provided)+ - **passed**: Number of expectations that passed+ - **total**: Total number of expectations+ - **pass_rate**: Fraction passed (0.0 to 1.0)+ - **details**: Individual expectation results++## Guidelines++- **Stay blind**: DO NOT try to infer which skill produced which output. Judge purely on output quality.+- **Be specific**: Cite specific examples when explaining strengths and weaknesses.+- **Be decisive**: Choose a winner unless outputs are genuinely equivalent.+- **Output quality first**: Assertion scores are secondary to overall task completion.+- **Be objective**: Don't favor outputs based on style preferences; focus on correctness and completeness.+- **Explain your reasoning**: The reasoning field should make it clear why you chose the winner.+- **Handle edge cases**: If both outputs fail, pick the one that fails less badly. If both are excellent, pick the one that's marginally better.diff --git a/.opencode/skills/skill-creator/agents/grader.md b/.opencode/skills/skill-creator/agents/grader.mdnew file mode 100644index 0000000..558ab05--- /dev/null+++ b/.opencode/skills/skill-creator/agents/grader.md@@ -0,0 +1,223 @@+# Grader Agent++Evaluate expectations against an execution transcript and outputs.++## Role++The Grader reviews a transcript and output files, then determines whether each expectation passes or fails. Provide clear evidence for each judgment.++You have two jobs: grade the outputs, and critique the evals themselves. A passing grade on a weak assertion is worse than useless — it creates false confidence. When you notice an assertion that's trivially satisfied, or an important outcome that no assertion checks, say so.++## Inputs++You receive these parameters in your prompt:++- **expectations**: List of expectations to evaluate (strings)+- **transcript_path**: Path to the execution transcript (markdown file)+- **outputs_dir**: Directory containing output files from execution++## Process++### Step 1: Read the Transcript++1. Read the transcript file completely+2. Note the eval prompt, execution steps, and final result+3. Identify any issues or errors documented++### Step 2: Examine Output Files++1. List files in outputs_dir+2. Read/examine each file relevant to the expectations. If outputs aren't plain text, use the inspection tools provided in your prompt — don't rely solely on what the transcript says the executor produced.+3. Note contents, structure, and quality++### Step 3: Evaluate Each Assertion++For each expectation:++1. **Search for evidence** in the transcript and outputs+2. **Determine verdict**:+ - **PASS**: Clear evidence the expectation is true AND the evidence reflects genuine task completion, not just surface-level compliance+ - **FAIL**: No evidence, or evidence contradicts the expectation, or the evidence is superficial (e.g., correct filename but empty/wrong content)+3. **Cite the evidence**: Quote the specific text or describe what you found++### Step 4: Extract and Verify Claims++Beyond the predefined expectations, extract implicit claims from the outputs and verify them:++1. **Extract claims** from the transcript and outputs:+ - Factual statements ("The form has 12 fields")+ - Process claims ("Used pypdf to fill the form")+ - Quality claims ("All fields were filled correctly")++2. **Verify each claim**:+ - **Factual claims**: Can be checked against the outputs or external sources+ - **Process claims**: Can be verified from the transcript+ - **Quality claims**: Evaluate whether the claim is justified++3. **Flag unverifiable claims**: Note claims that cannot be verified with available information++This catches issues that predefined expectations might miss.++### Step 5: Read User Notes++If `{outputs_dir}/user_notes.md` exists:+1. Read it and note any uncertainties or issues flagged by the executor+2. Include relevant concerns in the grading output+3. These may reveal problems even when expectations pass++### Step 6: Critique the Evals++After grading, consider whether the evals themselves could be improved. Only surface suggestions when there's a clear gap.++Good suggestions test meaningful outcomes — assertions that are hard to satisfy without actually doing the work correctly. Think about what makes an assertion *discriminating*: it passes when the skill genuinely succeeds and fails when it doesn't.++Suggestions worth raising:+- An assertion that passed but would also pass for a clearly wrong output (e.g., checking filename existence but not file content)+- An important outcome you observed — good or bad — that no assertion covers at all+- An assertion that can't actually be verified from the available outputs++Keep the bar high. The goal is to flag things the eval author would say "good catch" about, not to nitpick every assertion.++### Step 7: Write Grading Results++Save results to `{outputs_dir}/../grading.json` (sibling to outputs_dir).++## Grading Criteria++**PASS when**:+- The transcript or outputs clearly demonstrate the expectation is true+- Specific evidence can be cited+- The evidence reflects genuine substance, not just surface compliance (e.g., a file exists AND contains correct content, not just the right filename)++**FAIL when**:+- No evidence found for the expectation+- Evidence contradicts the expectation+- The expectation cannot be verified from available information+- The evidence is superficial — the assertion is technically satisfied but the underlying task outcome is wrong or incomplete+- The output appears to meet the assertion by coincidence rather than by actually doing the work++**When uncertain**: The burden of proof to pass is on the expectation.++### Step 8: Read Executor Metrics and Timing++1. If `{outputs_dir}/metrics.json` exists, read it and include in grading output+2. If `{outputs_dir}/../timing.json` exists, read it and include timing data++## Output Format++Write a JSON file with this structure:++```json+{+ "expectations": [+ {+ "text": "The output includes the name 'John Smith'",+ "passed": true,+ "evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'"+ },+ {+ "text": "The spreadsheet has a SUM formula in cell B10",+ "passed": false,+ "evidence": "No spreadsheet was created. The output was a text file."+ },+ {+ "text": "The assistant used the skill's OCR script",+ "passed": true,+ "evidence": "Transcript Step 2 shows: 'Tool: Bash - python ocr_script.py image.png'"+ }+ ],+ "summary": {+ "passed": 2,+ "failed": 1,+ "total": 3,+ "pass_rate": 0.67+ },+ "execution_metrics": {+ "tool_calls": {+ "Read": 5,+ "Write": 2,+ "Bash": 8+ },+ "total_tool_calls": 15,+ "total_steps": 6,+ "errors_encountered": 0,+ "output_chars": 12450,+ "transcript_chars": 3200+ },+ "timing": {+ "executor_duration_seconds": 165.0,+ "grader_duration_seconds": 26.0,+ "total_duration_seconds": 191.0+ },+ "claims": [+ {+ "claim": "The form has 12 fillable fields",+ "type": "factual",+ "verified": true,+ "evidence": "Counted 12 fields in field_info.json"+ },+ {+ "claim": "All required fields were populated",+ "type": "quality",+ "verified": false,+ "evidence": "Reference section was left blank despite data being available"+ }+ ],+ "user_notes_summary": {+ "uncertainties": ["Used 2023 data, may be stale"],+ "needs_review": [],+ "workarounds": ["Fell back to text overlay for non-fillable fields"]+ },+ "eval_feedback": {+ "suggestions": [+ {+ "assertion": "The output includes the name 'John Smith'",+ "reason": "A hallucinated document that mentions the name would also pass — consider checking it appears as the primary contact with matching phone and email from the input"+ },+ {+ "reason": "No assertion checks whether the extracted phone numbers match the input — I observed incorrect numbers in the output that went uncaught"+ }+ ],+ "overall": "Assertions check presence but not correctness. Consider adding content verification."+ }+}+```++## Field Descriptions++- **expectations**: Array of graded expectations+ - **text**: The original expectation text+ - **passed**: Boolean - true if expectation passes+ - **evidence**: Specific quote or description supporting the verdict+- **summary**: Aggregate statistics+ - **passed**: Count of passed expectations+ - **failed**: Count of failed expectations+ - **total**: Total expectations evaluated+ - **pass_rate**: Fraction passed (0.0 to 1.0)+- **execution_metrics**: Copied from executor's metrics.json (if available)+ - **output_chars**: Total character count of output files (proxy for tokens)+ - **transcript_chars**: Character count of transcript+- **timing**: Wall clock timing from timing.json (if available)+ - **executor_duration_seconds**: Time spent in executor subagent+ - **total_duration_seconds**: Total elapsed time for the run+- **claims**: Extracted and verified claims from the output+ - **claim**: The statement being verified+ - **type**: "factual", "process", or "quality"+ - **verified**: Boolean - whether the claim holds+ - **evidence**: Supporting or contradicting evidence+- **user_notes_summary**: Issues flagged by the executor+ - **uncertainties**: Things the executor wasn't sure about+ - **needs_review**: Items requiring human attention+ - **workarounds**: Places where the skill didn't work as expected+- **eval_feedback**: Improvement suggestions for the evals (only when warranted)+ - **suggestions**: List of concrete suggestions, each with a `reason` and optionally an `assertion` it relates to+ - **overall**: Brief assessment — can be "No suggestions, evals look solid" if nothing to flag++## Guidelines++- **Be objective**: Base verdicts on evidence, not assumptions+- **Be specific**: Quote the exact text that supports your verdict+- **Be thorough**: Check both transcript and output files+- **Be consistent**: Apply the same standard to each expectation+- **Explain failures**: Make it clear why evidence was insufficient+- **No partial credit**: Each expectation is pass or fail, not partialdiff --git a/.opencode/skills/skill-creator/assets/eval_review.html b/.opencode/skills/skill-creator/assets/eval_review.htmlnew file mode 100644index 0000000..938ff32--- /dev/null+++ b/.opencode/skills/skill-creator/assets/eval_review.html@@ -0,0 +1,146 @@+<!DOCTYPE html>+<html lang="en">+<head>+ <meta charset="UTF-8">+ <meta name="viewport" content="width=device-width, initial-scale=1.0">+ <title>Eval Set Review - __SKILL_NAME_PLACEHOLDER__</title>+ <link rel="preconnect" href="https://fonts.googleapis.com">+ <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>+ <link href="https://fonts.googleapis.com/css2?family=Poppins:wght@500;600&family=Lora:wght@400;500&display=swap" rel="stylesheet">+ <style>+ * { box-sizing: border-box; margin: 0; padding: 0; }+ body { font-family: 'Lora', Georgia, serif; background: #faf9f5; padding: 2rem; color: #141413; }+ h1 { font-family: 'Poppins', sans-serif; margin-bottom: 0.5rem; font-size: 1.5rem; }+ .description { color: #b0aea5; margin-bottom: 1.5rem; font-style: italic; max-width: 900px; }+ .controls { margin-bottom: 1rem; display: flex; gap: 0.5rem; }+ .btn { font-family: 'Poppins', sans-serif; padding: 0.5rem 1rem; border: none; border-radius: 6px; cursor: pointer; font-size: 0.875rem; font-weight: 500; }+ .btn-add { background: #6a9bcc; color: white; }+ .btn-add:hover { background: #5889b8; }+ .btn-export { background: #d97757; color: white; }+ .btn-export:hover { background: #c4613f; }+ table { width: 100%; max-width: 1100px; border-collapse: collapse; background: white; border-radius: 6px; overflow: hidden; box-shadow: 0 1px 3px rgba(0,0,0,0.08); }+ th { font-family: 'Poppins', sans-serif; background: #141413; color: #faf9f5; padding: 0.75rem 1rem; text-align: left; font-size: 0.875rem; }+ td { padding: 0.75rem 1rem; border-bottom: 1px solid #e8e6dc; vertical-align: top; }+ tr:nth-child(even) td { background: #faf9f5; }+ tr:hover td { background: #f3f1ea; }+ .section-header td { background: #e8e6dc; font-family: 'Poppins', sans-serif; font-weight: 500; font-size: 0.8rem; color: #141413; text-transform: uppercase; letter-spacing: 0.05em; }+ .query-input { width: 100%; padding: 0.4rem; border: 1px solid #e8e6dc; border-radius: 4px; font-size: 0.875rem; font-family: 'Lora', Georgia, serif; resize: vertical; min-height: 60px; }+ .query-input:focus { outline: none; border-color: #d97757; box-shadow: 0 0 0 2px rgba(217,119,87,0.15); }+ .toggle { position: relative; display: inline-block; width: 44px; height: 24px; }+ .toggle input { opacity: 0; width: 0; height: 0; }+ .toggle .slider { position: absolute; inset: 0; background: #b0aea5; border-radius: 24px; cursor: pointer; transition: 0.2s; }+ .toggle .slider::before { content: ""; position: absolute; width: 18px; height: 18px; left: 3px; bottom: 3px; background: white; border-radius: 50%; transition: 0.2s; }+ .toggle input:checked + .slider { background: #d97757; }+ .toggle input:checked + .slider::before { transform: translateX(20px); }+ .btn-delete { background: #c44; color: white; padding: 0.3rem 0.6rem; border: none; border-radius: 4px; cursor: pointer; font-size: 0.75rem; font-family: 'Poppins', sans-serif; }+ .btn-delete:hover { background: #a33; }+ .summary { margin-top: 1rem; color: #b0aea5; font-size: 0.875rem; }+ </style>+</head>+<body>+ <h1>Eval Set Review: <span id="skill-name">__SKILL_NAME_PLACEHOLDER__</span></h1>+ <p class="description">Current description: <span id="skill-desc">__SKILL_DESCRIPTION_PLACEHOLDER__</span></p>++ <div class="controls">+ <button class="btn btn-add" onclick="addRow()">+ Add Query</button>+ <button class="btn btn-export" onclick="exportEvalSet()">Export Eval Set</button>+ </div>++ <table>+ <thead>+ <tr>+ <th style="width:65%">Query</th>+ <th style="width:18%">Should Trigger</th>+ <th style="width:10%">Actions</th>+ </tr>+ </thead>+ <tbody id="eval-body"></tbody>+ </table>++ <p class="summary" id="summary"></p>++ <script>+ const EVAL_DATA = __EVAL_DATA_PLACEHOLDER__;++ let evalItems = [...EVAL_DATA];++ function render() {+ const tbody = document.getElementById('eval-body');+ tbody.innerHTML = '';++ // Sort: should-trigger first, then should-not-trigger+ const sorted = evalItems+ .map((item, origIdx) => ({ ...item, origIdx }))+ .sort((a, b) => (b.should_trigger ? 1 : 0) - (a.should_trigger ? 1 : 0));++ let lastGroup = null;+ sorted.forEach(item => {+ const group = item.should_trigger ? 'trigger' : 'no-trigger';+ if (group !== lastGroup) {+ const headerRow = document.createElement('tr');+ headerRow.className = 'section-header';+ headerRow.innerHTML = `<td colspan="3">${item.should_trigger ? 'Should Trigger' : 'Should NOT Trigger'}</td>`;+ tbody.appendChild(headerRow);+ lastGroup = group;+ }++ const idx = item.origIdx;+ const tr = document.createElement('tr');+ tr.innerHTML = `+ <td><textarea class="query-input" onchange="updateQuery(${idx}, this.value)">${escapeHtml(item.query)}</textarea></td>+ <td>+ <label class="toggle">+ <input type="checkbox" ${item.should_trigger ? 'checked' : ''} onchange="updateTrigger(${idx}, this.checked)">+ <span class="slider"></span>+ </label>+ <span style="margin-left:8px;font-size:0.8rem;color:#b0aea5">${item.should_trigger ? 'Yes' : 'No'}</span>+ </td>+ <td><button class="btn-delete" onclick="deleteRow(${idx})">Delete</button></td>+ `;+ tbody.appendChild(tr);+ });+ updateSummary();+ }++ function escapeHtml(text) {+ const div = document.createElement('div');+ div.textContent = text;+ return div.innerHTML;+ }++ function updateQuery(idx, value) { evalItems[idx].query = value; updateSummary(); }+ function updateTrigger(idx, value) { evalItems[idx].should_trigger = value; render(); }+ function deleteRow(idx) { evalItems.splice(idx, 1); render(); }++ function addRow() {+ evalItems.push({ query: '', should_trigger: true });+ render();+ const inputs = document.querySelectorAll('.query-input');+ inputs[inputs.length - 1].focus();+ }++ function updateSummary() {+ const trigger = evalItems.filter(i => i.should_trigger).length;+ const noTrigger = evalItems.filter(i => !i.should_trigger).length;+ document.getElementById('summary').textContent =+ `${evalItems.length} queries total: ${trigger} should trigger, ${noTrigger} should not trigger`;+ }++ function exportEvalSet() {+ const valid = evalItems.filter(i => i.query.trim() !== '');+ const data = valid.map(i => ({ query: i.query.trim(), should_trigger: i.should_trigger }));+ const blob = new Blob([JSON.stringify(data, null, 2)], { type: 'application/json' });+ const url = URL.createObjectURL(blob);+ const a = document.createElement('a');+ a.href = url;+ a.download = 'eval_set.json';+ document.body.appendChild(a);+ a.click();+ document.body.removeChild(a);+ URL.revokeObjectURL(url);+ }++ render();+ </script>+</body>+</html>diff --git a/.opencode/skills/skill-creator/eval-viewer/generate_review.py b/.opencode/skills/skill-creator/eval-viewer/generate_review.pynew file mode 100644index 0000000..7fa5978--- /dev/null+++ b/.opencode/skills/skill-creator/eval-viewer/generate_review.py@@ -0,0 +1,471 @@+#!/usr/bin/env python3+"""Generate and serve a review page for eval results.++Reads the workspace directory, discovers runs (directories with outputs/),+embeds all output data into a self-contained HTML page, and serves it via+a tiny HTTP server. Feedback auto-saves to feedback.json in the workspace.++Usage:+ python generate_review.py <workspace-path> [--port PORT] [--skill-name NAME]+ python generate_review.py <workspace-path> --previous-feedback /path/to/old/feedback.json++No dependencies beyond the Python stdlib are required.+"""++import argparse+import base64+import json+import mimetypes+import os+import re+import signal+import subprocess+import sys+import time+import webbrowser+from functools import partial+from http.server import HTTPServer, BaseHTTPRequestHandler+from pathlib import Path++# Files to exclude from output listings+METADATA_FILES = {"transcript.md", "user_notes.md", "metrics.json"}++# Extensions we render as inline text+TEXT_EXTENSIONS = {+ ".txt", ".md", ".json", ".csv", ".py", ".js", ".ts", ".tsx", ".jsx",+ ".yaml", ".yml", ".xml", ".html", ".css", ".sh", ".rb", ".go", ".rs",+ ".java", ".c", ".cpp", ".h", ".hpp", ".sql", ".r", ".toml",+}++# Extensions we render as inline images+IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp"}++# MIME type overrides for common types+MIME_OVERRIDES = {+ ".svg": "image/svg+xml",+ ".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",+ ".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",+ ".pptx": "application/vnd.openxmlformats-officedocument.presentationml.presentation",+}+++def get_mime_type(path: Path) -> str:+ ext = path.suffix.lower()+ if ext in MIME_OVERRIDES:+ return MIME_OVERRIDES[ext]+ mime, _ = mimetypes.guess_type(str(path))+ return mime or "application/octet-stream"+++def find_runs(workspace: Path) -> list[dict]:+ """Recursively find directories that contain an outputs/ subdirectory."""+ runs: list[dict] = []+ _find_runs_recursive(workspace, workspace, runs)+ runs.sort(key=lambda r: (r.get("eval_id", float("inf")), r["id"]))+ return runs+++def _find_runs_recursive(root: Path, current: Path, runs: list[dict]) -> None:+ if not current.is_dir():+ return++ outputs_dir = current / "outputs"+ if outputs_dir.is_dir():+ run = build_run(root, current)+ if run:+ runs.append(run)+ return++ skip = {"node_modules", ".git", "__pycache__", "skill", "inputs"}+ for child in sorted(current.iterdir()):+ if child.is_dir() and child.name not in skip:+ _find_runs_recursive(root, child, runs)+++def build_run(root: Path, run_dir: Path) -> dict | None:+ """Build a run dict with prompt, outputs, and grading data."""+ prompt = ""+ eval_id = None++ # Try eval_metadata.json+ for candidate in [run_dir / "eval_metadata.json", run_dir.parent / "eval_metadata.json"]:+ if candidate.exists():+ try:+ metadata = json.loads(candidate.read_text())+ prompt = metadata.get("prompt", "")+ eval_id = metadata.get("eval_id")+ except (json.JSONDecodeError, OSError):+ pass+ if prompt:+ break++ # Fall back to transcript.md+ if not prompt:+ for candidate in [run_dir / "transcript.md", run_dir / "outputs" / "transcript.md"]:+ if candidate.exists():+ try:+ text = candidate.read_text()+ match = re.search(r"## Eval Prompt\n\n([\s\S]*?)(?=\n##|$)", text)+ if match:+ prompt = match.group(1).strip()+ except OSError:+ pass+ if prompt:+ break++ if not prompt:+ prompt = "(No prompt found)"++ run_id = str(run_dir.relative_to(root)).replace("/", "-").replace("\\", "-")++ # Collect output files+ outputs_dir = run_dir / "outputs"+ output_files: list[dict] = []+ if outputs_dir.is_dir():+ for f in sorted(outputs_dir.iterdir()):+ if f.is_file() and f.name not in METADATA_FILES:+ output_files.append(embed_file(f))++ # Load grading if present+ grading = None+ for candidate in [run_dir / "grading.json", run_dir.parent / "grading.json"]:+ if candidate.exists():+ try:+ grading = json.loads(candidate.read_text())+ except (json.JSONDecodeError, OSError):+ pass+ if grading:+ break++ return {+ "id": run_id,+ "prompt": prompt,+ "eval_id": eval_id,+ "outputs": output_files,+ "grading": grading,+ }+++def embed_file(path: Path) -> dict:+ """Read a file and return an embedded representation."""+ ext = path.suffix.lower()+ mime = get_mime_type(path)++ if ext in TEXT_EXTENSIONS:+ try:+ content = path.read_text(errors="replace")+ except OSError:+ content = "(Error reading file)"+ return {+ "name": path.name,+ "type": "text",+ "content": content,+ }+ elif ext in IMAGE_EXTENSIONS:+ try:+ raw = path.read_bytes()+ b64 = base64.b64encode(raw).decode("ascii")+ except OSError:+ return {"name": path.name, "type": "error", "content": "(Error reading file)"}+ return {+ "name": path.name,+ "type": "image",+ "mime": mime,+ "data_uri": f"data:{mime};base64,{b64}",+ }+ elif ext == ".pdf":+ try:+ raw = path.read_bytes()+ b64 = base64.b64encode(raw).decode("ascii")+ except OSError:+ return {"name": path.name, "type": "error", "content": "(Error reading file)"}+ return {+ "name": path.name,+ "type": "pdf",+ "data_uri": f"data:{mime};base64,{b64}",+ }+ elif ext == ".xlsx":+ try:+ raw = path.read_bytes()+ b64 = base64.b64encode(raw).decode("ascii")+ except OSError:+ return {"name": path.name, "type": "error", "content": "(Error reading file)"}+ return {+ "name": path.name,+ "type": "xlsx",+ "data_b64": b64,+ }+ else:+ # Binary / unknown — base64 download link+ try:+ raw = path.read_bytes()+ b64 = base64.b64encode(raw).decode("ascii")+ except OSError:+ return {"name": path.name, "type": "error", "content": "(Error reading file)"}+ return {+ "name": path.name,+ "type": "binary",+ "mime": mime,+ "data_uri": f"data:{mime};base64,{b64}",+ }+++def load_previous_iteration(workspace: Path) -> dict[str, dict]:+ """Load previous iteration's feedback and outputs.++ Returns a map of run_id -> {"feedback": str, "outputs": list[dict]}.+ """+ result: dict[str, dict] = {}++ # Load feedback+ feedback_map: dict[str, str] = {}+ feedback_path = workspace / "feedback.json"+ if feedback_path.exists():+ try:+ data = json.loads(feedback_path.read_text())+ feedback_map = {+ r["run_id"]: r["feedback"]+ for r in data.get("reviews", [])+ if r.get("feedback", "").strip()+ }+ except (json.JSONDecodeError, OSError, KeyError):+ pass++ # Load runs (to get outputs)+ prev_runs = find_runs(workspace)+ for run in prev_runs:+ result[run["id"]] = {+ "feedback": feedback_map.get(run["id"], ""),+ "outputs": run.get("outputs", []),+ }++ # Also add feedback for run_ids that had feedback but no matching run+ for run_id, fb in feedback_map.items():+ if run_id not in result:+ result[run_id] = {"feedback": fb, "outputs": []}++ return result+++def generate_html(+ runs: list[dict],+ skill_name: str,+ previous: dict[str, dict] | None = None,+ benchmark: dict | None = None,+) -> str:+ """Generate the complete standalone HTML page with embedded data."""+ template_path = Path(__file__).parent / "viewer.html"+ template = template_path.read_text()++ # Build previous_feedback and previous_outputs maps for the template+ previous_feedback: dict[str, str] = {}+ previous_outputs: dict[str, list[dict]] = {}+ if previous:+ for run_id, data in previous.items():+ if data.get("feedback"):+ previous_feedback[run_id] = data["feedback"]+ if data.get("outputs"):+ previous_outputs[run_id] = data["outputs"]++ embedded = {+ "skill_name": skill_name,+ "runs": runs,+ "previous_feedback": previous_feedback,+ "previous_outputs": previous_outputs,+ }+ if benchmark:+ embedded["benchmark"] = benchmark++ data_json = json.dumps(embedded)++ return template.replace("/*__EMBEDDED_DATA__*/", f"const EMBEDDED_DATA = {data_json};")+++# ---------------------------------------------------------------------------+# HTTP server (stdlib only, zero dependencies)+# ---------------------------------------------------------------------------++def _kill_port(port: int) -> None:+ """Kill any process listening on the given port."""+ try:+ result = subprocess.run(+ ["lsof", "-ti", f":{port}"],+ capture_output=True, text=True, timeout=5,+ )+ for pid_str in result.stdout.strip().split("\n"):+ if pid_str.strip():+ try:+ os.kill(int(pid_str.strip()), signal.SIGTERM)+ except (ProcessLookupError, ValueError):+ pass+ if result.stdout.strip():+ time.sleep(0.5)+ except subprocess.TimeoutExpired:+ pass+ except FileNotFoundError:+ print("Note: lsof not found, cannot check if port is in use", file=sys.stderr)++class ReviewHandler(BaseHTTPRequestHandler):+ """Serves the review HTML and handles feedback saves.++ Regenerates the HTML on each page load so that refreshing the browser+ picks up new eval outputs without restarting the server.+ """++ def __init__(+ self,+ workspace: Path,+ skill_name: str,+ feedback_path: Path,+ previous: dict[str, dict],+ benchmark_path: Path | None,+ *args,+ **kwargs,+ ):+ self.workspace = workspace+ self.skill_name = skill_name+ self.feedback_path = feedback_path+ self.previous = previous+ self.benchmark_path = benchmark_path+ super().__init__(*args, **kwargs)++ def do_GET(self) -> None:+ if self.path == "/" or self.path == "/index.html":+ # Regenerate HTML on each request (re-scans workspace for new outputs)+ runs = find_runs(self.workspace)+ benchmark = None+ if self.benchmark_path and self.benchmark_path.exists():+ try:+ benchmark = json.loads(self.benchmark_path.read_text())+ except (json.JSONDecodeError, OSError):+ pass+ html = generate_html(runs, self.skill_name, self.previous, benchmark)+ content = html.encode("utf-8")+ self.send_response(200)+ self.send_header("Content-Type", "text/html; charset=utf-8")+ self.send_header("Content-Length", str(len(content)))+ self.end_headers()+ self.wfile.write(content)+ elif self.path == "/api/feedback":+ data = b"{}"+ if self.feedback_path.exists():+ data = self.feedback_path.read_bytes()+ self.send_response(200)+ self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(data)))+ self.end_headers()+ self.wfile.write(data)+ else:+ self.send_error(404)++ def do_POST(self) -> None:+ if self.path == "/api/feedback":+ length = int(self.headers.get("Content-Length", 0))+ body = self.rfile.read(length)+ try:+ data = json.loads(body)+ if not isinstance(data, dict) or "reviews" not in data:+ raise ValueError("Expected JSON object with 'reviews' key")+ self.feedback_path.write_text(json.dumps(data, indent=2) + "\n")+ resp = b'{"ok":true}'+ self.send_response(200)+ except (json.JSONDecodeError, OSError, ValueError) as e:+ resp = json.dumps({"error": str(e)}).encode()+ self.send_response(500)+ self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(resp)))+ self.end_headers()+ self.wfile.write(resp)+ else:+ self.send_error(404)++ def log_message(self, format: str, *args: object) -> None:+ # Suppress request logging to keep terminal clean+ pass+++def main() -> None:+ parser = argparse.ArgumentParser(description="Generate and serve eval review")+ parser.add_argument("workspace", type=Path, help="Path to workspace directory")+ parser.add_argument("--port", "-p", type=int, default=3117, help="Server port (default: 3117)")+ parser.add_argument("--skill-name", "-n", type=str, default=None, help="Skill name for header")+ parser.add_argument(+ "--previous-workspace", type=Path, default=None,+ help="Path to previous iteration's workspace (shows old outputs and feedback as context)",+ )+ parser.add_argument(+ "--benchmark", type=Path, default=None,+ help="Path to benchmark.json to show in the Benchmark tab",+ )+ parser.add_argument(+ "--static", "-s", type=Path, default=None,+ help="Write standalone HTML to this path instead of starting a server",+ )+ args = parser.parse_args()++ workspace = args.workspace.resolve()+ if not workspace.is_dir():+ print(f"Error: {workspace} is not a directory", file=sys.stderr)+ sys.exit(1)++ runs = find_runs(workspace)+ if not runs:+ print(f"No runs found in {workspace}", file=sys.stderr)+ sys.exit(1)++ skill_name = args.skill_name or workspace.name.replace("-workspace", "")+ feedback_path = workspace / "feedback.json"++ previous: dict[str, dict] = {}+ if args.previous_workspace:+ previous = load_previous_iteration(args.previous_workspace.resolve())++ benchmark_path = args.benchmark.resolve() if args.benchmark else None+ benchmark = None+ if benchmark_path and benchmark_path.exists():+ try:+ benchmark = json.loads(benchmark_path.read_text())+ except (json.JSONDecodeError, OSError):+ pass++ if args.static:+ html = generate_html(runs, skill_name, previous, benchmark)+ args.static.parent.mkdir(parents=True, exist_ok=True)+ args.static.write_text(html)+ print(f"\n Static viewer written to: {args.static}\n")+ sys.exit(0)++ # Kill any existing process on the target port+ port = args.port+ _kill_port(port)+ handler = partial(ReviewHandler, workspace, skill_name, feedback_path, previous, benchmark_path)+ try:+ server = HTTPServer(("127.0.0.1", port), handler)+ except OSError:+ # Port still in use after kill attempt — find a free one+ server = HTTPServer(("127.0.0.1", 0), handler)+ port = server.server_address[1]++ url = f"http://localhost:{port}"+ print(f"\n Eval Viewer")+ print(f" ─────────────────────────────────")+ print(f" URL: {url}")+ print(f" Workspace: {workspace}")+ print(f" Feedback: {feedback_path}")+ if previous:+ print(f" Previous: {args.previous_workspace} ({len(previous)} runs)")+ if benchmark_path:+ print(f" Benchmark: {benchmark_path}")+ print(f"\n Press Ctrl+C to stop.\n")++ webbrowser.open(url)++ try:+ server.serve_forever()+ except KeyboardInterrupt:+ print("\nStopped.")+ server.server_close()+++if __name__ == "__main__":+ main()diff --git a/.opencode/skills/skill-creator/eval-viewer/viewer.html b/.opencode/skills/skill-creator/eval-viewer/viewer.htmlnew file mode 100644index 0000000..6d8e963--- /dev/null+++ b/.opencode/skills/skill-creator/eval-viewer/viewer.html@@ -0,0 +1,1325 @@+<!DOCTYPE html>+<html lang="en">+<head>+ <meta charset="UTF-8">+ <meta name="viewport" content="width=device-width, initial-scale=1.0">+ <title>Eval Review</title>+ <link rel="preconnect" href="https://fonts.googleapis.com">+ <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>+ <link href="https://fonts.googleapis.com/css2?family=Poppins:wght@500;600&family=Lora:wght@400;500&display=swap" rel="stylesheet">+ <script src="https://cdn.sheetjs.com/xlsx-0.20.3/package/dist/xlsx.full.min.js" integrity="sha384-EnyY0/GSHQGSxSgMwaIPzSESbqoOLSexfnSMN2AP+39Ckmn92stwABZynq1JyzdT" crossorigin="anonymous"></script>+ <style>+ :root {+ --bg: #faf9f5;+ --surface: #ffffff;+ --border: #e8e6dc;+ --text: #141413;+ --text-muted: #b0aea5;+ --accent: #d97757;+ --accent-hover: #c4613f;+ --green: #788c5d;+ --green-bg: #eef2e8;+ --red: #c44;+ --red-bg: #fceaea;+ --header-bg: #141413;+ --header-text: #faf9f5;+ --radius: 6px;+ }++ * { box-sizing: border-box; margin: 0; padding: 0; }++ body {+ font-family: 'Lora', Georgia, serif;+ background: var(--bg);+ color: var(--text);+ height: 100vh;+ display: flex;+ flex-direction: column;+ }++ /* ---- Header ---- */+ .header {+ background: var(--header-bg);+ color: var(--header-text);+ padding: 1rem 2rem;+ display: flex;+ justify-content: space-between;+ align-items: center;+ flex-shrink: 0;+ }+ .header h1 {+ font-family: 'Poppins', sans-serif;+ font-size: 1.25rem;+ font-weight: 600;+ }+ .header .instructions {+ font-size: 0.8rem;+ opacity: 0.7;+ margin-top: 0.25rem;+ }+ .header .progress {+ font-size: 0.875rem;+ opacity: 0.8;+ text-align: right;+ }++ /* ---- Main content ---- */+ .main {+ flex: 1;+ overflow-y: auto;+ padding: 1.5rem 2rem;+ display: flex;+ flex-direction: column;+ gap: 1.25rem;+ }++ /* ---- Sections ---- */+ .section {+ background: var(--surface);+ border: 1px solid var(--border);+ border-radius: var(--radius);+ flex-shrink: 0;+ }+ .section-header {+ font-family: 'Poppins', sans-serif;+ padding: 0.75rem 1rem;+ font-size: 0.75rem;+ font-weight: 500;+ text-transform: uppercase;+ letter-spacing: 0.05em;+ color: var(--text-muted);+ border-bottom: 1px solid var(--border);+ background: var(--bg);+ }+ .section-body {+ padding: 1rem;+ }++ /* ---- Config badge ---- */+ .config-badge {+ display: inline-block;+ padding: 0.2rem 0.625rem;+ border-radius: 9999px;+ font-family: 'Poppins', sans-serif;+ font-size: 0.6875rem;+ font-weight: 600;+ text-transform: uppercase;+ letter-spacing: 0.03em;+ margin-left: 0.75rem;+ vertical-align: middle;+ }+ .config-badge.config-primary {+ background: rgba(33, 150, 243, 0.12);+ color: #1976d2;+ }+ .config-badge.config-baseline {+ background: rgba(255, 193, 7, 0.15);+ color: #f57f17;+ }++ /* ---- Prompt ---- */+ .prompt-text {+ white-space: pre-wrap;+ font-size: 0.9375rem;+ line-height: 1.6;+ }++ /* ---- Outputs ---- */+ .output-file {+ border: 1px solid var(--border);+ border-radius: var(--radius);+ overflow: hidden;+ }+ .output-file + .output-file {+ margin-top: 1rem;+ }+ .output-file-header {+ padding: 0.5rem 0.75rem;+ font-size: 0.8rem;+ font-weight: 600;+ color: var(--text-muted);+ background: var(--bg);+ border-bottom: 1px solid var(--border);+ font-family: 'SF Mono', SFMono-Regular, Consolas, 'Liberation Mono', Menlo, monospace;+ display: flex;+ justify-content: space-between;+ align-items: center;+ }+ .output-file-header .dl-btn {+ font-size: 0.7rem;+ color: var(--accent);+ text-decoration: none;+ cursor: pointer;+ font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;+ font-weight: 500;+ opacity: 0.8;+ }+ .output-file-header .dl-btn:hover {+ opacity: 1;+ text-decoration: underline;+ }+ .output-file-content {+ padding: 0.75rem;+ overflow-x: auto;+ }+ .output-file-content pre {+ font-size: 0.8125rem;+ line-height: 1.5;+ white-space: pre-wrap;+ word-break: break-word;+ font-family: 'SF Mono', SFMono-Regular, Consolas, 'Liberation Mono', Menlo, monospace;+ }+ .output-file-content img {+ max-width: 100%;+ height: auto;+ border-radius: 4px;+ }+ .output-file-content iframe {+ width: 100%;+ height: 600px;+ border: none;+ }+ .output-file-content table {+ border-collapse: collapse;+ font-size: 0.8125rem;+ width: 100%;+ }+ .output-file-content table td,+ .output-file-content table th {+ border: 1px solid var(--border);+ padding: 0.375rem 0.5rem;+ text-align: left;+ }+ .output-file-content table th {+ background: var(--bg);+ font-weight: 600;+ }+ .output-file-content .download-link {+ display: inline-flex;+ align-items: center;+ gap: 0.5rem;+ padding: 0.5rem 1rem;+ background: var(--bg);+ border: 1px solid var(--border);+ border-radius: 4px;+ color: var(--accent);+ text-decoration: none;+ font-size: 0.875rem;+ cursor: pointer;+ }+ .output-file-content .download-link:hover {+ background: var(--border);+ }+ .empty-state {+ color: var(--text-muted);+ font-style: italic;+ padding: 2rem;+ text-align: center;+ }++ /* ---- Feedback ---- */+ .prev-feedback {+ background: var(--bg);+ border: 1px solid var(--border);+ border-radius: 4px;+ padding: 0.625rem 0.75rem;+ margin-top: 0.75rem;+ font-size: 0.8125rem;+ color: var(--text-muted);+ line-height: 1.5;+ }+ .prev-feedback-label {+ font-size: 0.7rem;+ font-weight: 600;+ text-transform: uppercase;+ letter-spacing: 0.04em;+ margin-bottom: 0.25rem;+ color: var(--text-muted);+ }+ .feedback-textarea {+ width: 100%;+ min-height: 100px;+ padding: 0.75rem;+ border: 1px solid var(--border);+ border-radius: 4px;+ font-family: inherit;+ font-size: 0.9375rem;+ line-height: 1.5;+ resize: vertical;+ color: var(--text);+ }+ .feedback-textarea:focus {+ outline: none;+ border-color: var(--accent);+ box-shadow: 0 0 0 3px rgba(37, 99, 235, 0.1);+ }+ .feedback-status {+ font-size: 0.75rem;+ color: var(--text-muted);+ margin-top: 0.5rem;+ min-height: 1.1em;+ }++ /* ---- Grades (collapsible) ---- */+ .grades-toggle {+ display: flex;+ align-items: center;+ cursor: pointer;+ user-select: none;+ }+ .grades-toggle:hover {+ color: var(--accent);+ }+ .grades-toggle .arrow {+ margin-right: 0.5rem;+ transition: transform 0.15s;+ font-size: 0.75rem;+ }+ .grades-toggle .arrow.open {+ transform: rotate(90deg);+ }+ .grades-content {+ display: none;+ margin-top: 0.75rem;+ }+ .grades-content.open {+ display: block;+ }+ .grades-summary {+ font-size: 0.875rem;+ margin-bottom: 0.75rem;+ display: flex;+ align-items: center;+ gap: 0.5rem;+ }+ .grade-badge {+ display: inline-block;+ padding: 0.125rem 0.5rem;+ border-radius: 9999px;+ font-size: 0.75rem;+ font-weight: 600;+ }+ .grade-pass { background: var(--green-bg); color: var(--green); }+ .grade-fail { background: var(--red-bg); color: var(--red); }+ .assertion-list {+ list-style: none;+ }+ .assertion-item {+ padding: 0.625rem 0;+ border-bottom: 1px solid var(--border);+ font-size: 0.8125rem;+ }+ .assertion-item:last-child { border-bottom: none; }+ .assertion-status {+ font-weight: 600;+ margin-right: 0.5rem;+ }+ .assertion-status.pass { color: var(--green); }+ .assertion-status.fail { color: var(--red); }+ .assertion-evidence {+ color: var(--text-muted);+ font-size: 0.75rem;+ margin-top: 0.25rem;+ padding-left: 1.5rem;+ }++ /* ---- View tabs ---- */+ .view-tabs {+ display: flex;+ gap: 0;+ padding: 0 2rem;+ background: var(--bg);+ border-bottom: 1px solid var(--border);+ flex-shrink: 0;+ }+ .view-tab {+ font-family: 'Poppins', sans-serif;+ padding: 0.625rem 1.25rem;+ font-size: 0.8125rem;+ font-weight: 500;+ cursor: pointer;+ border: none;+ background: none;+ color: var(--text-muted);+ border-bottom: 2px solid transparent;+ transition: all 0.15s;+ }+ .view-tab:hover { color: var(--text); }+ .view-tab.active {+ color: var(--accent);+ border-bottom-color: var(--accent);+ }+ .view-panel { display: none; }+ .view-panel.active { display: flex; flex-direction: column; flex: 1; overflow: hidden; }++ /* ---- Benchmark view ---- */+ .benchmark-view {+ padding: 1.5rem 2rem;+ overflow-y: auto;+ flex: 1;+ }+ .benchmark-table {+ border-collapse: collapse;+ background: var(--surface);+ border: 1px solid var(--border);+ border-radius: var(--radius);+ font-size: 0.8125rem;+ width: 100%;+ margin-bottom: 1.5rem;+ }+ .benchmark-table th, .benchmark-table td {+ padding: 0.625rem 0.75rem;+ text-align: left;+ border: 1px solid var(--border);+ }+ .benchmark-table th {+ font-family: 'Poppins', sans-serif;+ background: var(--header-bg);+ color: var(--header-text);+ font-weight: 500;+ font-size: 0.75rem;+ text-transform: uppercase;+ letter-spacing: 0.04em;+ }+ .benchmark-table tr:hover { background: var(--bg); }+ .benchmark-table tr.benchmark-row-with { background: rgba(33, 150, 243, 0.06); }+ .benchmark-table tr.benchmark-row-without { background: rgba(255, 193, 7, 0.06); }+ .benchmark-table tr.benchmark-row-with:hover { background: rgba(33, 150, 243, 0.12); }+ .benchmark-table tr.benchmark-row-without:hover { background: rgba(255, 193, 7, 0.12); }+ .benchmark-table tr.benchmark-row-avg { font-weight: 600; border-top: 2px solid var(--border); }+ .benchmark-table tr.benchmark-row-avg.benchmark-row-with { background: rgba(33, 150, 243, 0.12); }+ .benchmark-table tr.benchmark-row-avg.benchmark-row-without { background: rgba(255, 193, 7, 0.12); }+ .benchmark-delta-positive { color: var(--green); font-weight: 600; }+ .benchmark-delta-negative { color: var(--red); font-weight: 600; }+ .benchmark-notes {+ background: var(--surface);+ border: 1px solid var(--border);+ border-radius: var(--radius);+ padding: 1rem;+ }+ .benchmark-notes h3 {+ font-family: 'Poppins', sans-serif;+ font-size: 0.875rem;+ margin-bottom: 0.75rem;+ }+ .benchmark-notes ul {+ list-style: disc;+ padding-left: 1.25rem;+ }+ .benchmark-notes li {+ font-size: 0.8125rem;+ line-height: 1.6;+ margin-bottom: 0.375rem;+ }+ .benchmark-empty {+ color: var(--text-muted);+ font-style: italic;+ text-align: center;+ padding: 3rem;+ }++ /* ---- Navigation ---- */+ .nav {+ display: flex;+ justify-content: space-between;+ align-items: center;+ padding: 1rem 2rem;+ border-top: 1px solid var(--border);+ background: var(--surface);+ flex-shrink: 0;+ }+ .nav-btn {+ font-family: 'Poppins', sans-serif;+ padding: 0.5rem 1.25rem;+ border: 1px solid var(--border);+ border-radius: var(--radius);+ background: var(--surface);+ cursor: pointer;+ font-size: 0.875rem;+ font-weight: 500;+ color: var(--text);+ transition: all 0.15s;+ }+ .nav-btn:hover:not(:disabled) {+ background: var(--bg);+ border-color: var(--text-muted);+ }+ .nav-btn:disabled {+ opacity: 0.4;+ cursor: not-allowed;+ }+ .done-btn {+ font-family: 'Poppins', sans-serif;+ padding: 0.5rem 1.5rem;+ border: 1px solid var(--border);+ border-radius: var(--radius);+ background: var(--surface);+ color: var(--text);+ cursor: pointer;+ font-size: 0.875rem;+ font-weight: 500;+ transition: all 0.15s;+ }+ .done-btn:hover {+ background: var(--bg);+ border-color: var(--text-muted);+ }+ .done-btn.ready {+ border: none;+ background: var(--accent);+ color: white;+ font-weight: 600;+ }+ .done-btn.ready:hover {+ background: var(--accent-hover);+ }+ /* ---- Done overlay ---- */+ .done-overlay {+ display: none;+ position: fixed;+ inset: 0;+ background: rgba(0, 0, 0, 0.5);+ z-index: 100;+ justify-content: center;+ align-items: center;+ }+ .done-overlay.visible {+ display: flex;+ }+ .done-card {+ background: var(--surface);+ border-radius: 12px;+ padding: 2rem 3rem;+ text-align: center;+ box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);+ max-width: 500px;+ }+ .done-card h2 {+ font-size: 1.5rem;+ margin-bottom: 0.5rem;+ }+ .done-card p {+ color: var(--text-muted);+ margin-bottom: 1.5rem;+ line-height: 1.5;+ }+ .done-card .btn-row {+ display: flex;+ gap: 0.5rem;+ justify-content: center;+ }+ .done-card button {+ padding: 0.5rem 1.25rem;+ border: 1px solid var(--border);+ border-radius: var(--radius);+ background: var(--surface);+ cursor: pointer;+ font-size: 0.875rem;+ }+ .done-card button:hover {+ background: var(--bg);+ }+ /* ---- Toast ---- */+ .toast {+ position: fixed;+ bottom: 5rem;+ left: 50%;+ transform: translateX(-50%);+ background: var(--header-bg);+ color: var(--header-text);+ padding: 0.625rem 1.25rem;+ border-radius: var(--radius);+ font-size: 0.875rem;+ opacity: 0;+ transition: opacity 0.3s;+ pointer-events: none;+ z-index: 200;+ }+ .toast.visible {+ opacity: 1;+ }+ </style>+</head>+<body>+ <div id="app" style="height:100vh; display:flex; flex-direction:column;">+ <div class="header">+ <div>+ <h1>Eval Review: <span id="skill-name"></span></h1>+ <div class="instructions">Review each output and leave feedback below. Navigate with arrow keys or buttons. When done, copy feedback and paste into Claude Code.</div>+ </div>+ <div class="progress" id="progress"></div>+ </div>++ <!-- View tabs (only shown when benchmark data exists) -->+ <div class="view-tabs" id="view-tabs" style="display:none;">+ <button class="view-tab active" onclick="switchView('outputs')">Outputs</button>+ <button class="view-tab" onclick="switchView('benchmark')">Benchmark</button>+ </div>++ <!-- Outputs panel (qualitative review) -->+ <div class="view-panel active" id="panel-outputs">+ <div class="main">+ <!-- Prompt -->+ <div class="section">+ <div class="section-header">Prompt <span class="config-badge" id="config-badge" style="display:none;"></span></div>+ <div class="section-body">+ <div class="prompt-text" id="prompt-text"></div>+ </div>+ </div>++ <!-- Outputs -->+ <div class="section">+ <div class="section-header">Output</div>+ <div class="section-body" id="outputs-body">+ <div class="empty-state">No output files found</div>+ </div>+ </div>++ <!-- Previous Output (collapsible) -->+ <div class="section" id="prev-outputs-section" style="display:none;">+ <div class="section-header">+ <div class="grades-toggle" onclick="togglePrevOutputs()">+ <span class="arrow" id="prev-outputs-arrow">▶</span>+ Previous Output+ </div>+ </div>+ <div class="grades-content" id="prev-outputs-content"></div>+ </div>++ <!-- Grades (collapsible) -->+ <div class="section" id="grades-section" style="display:none;">+ <div class="section-header">+ <div class="grades-toggle" onclick="toggleGrades()">+ <span class="arrow" id="grades-arrow">▶</span>+ Formal Grades+ </div>+ </div>+ <div class="grades-content" id="grades-content"></div>+ </div>++ <!-- Feedback -->+ <div class="section">+ <div class="section-header">Your Feedback</div>+ <div class="section-body">+ <textarea+ class="feedback-textarea"+ id="feedback"+ placeholder="What do you think of this output? Any issues, suggestions, or things that look great?"+ ></textarea>+ <div class="feedback-status" id="feedback-status"></div>+ <div class="prev-feedback" id="prev-feedback" style="display:none;">+ <div class="prev-feedback-label">Previous feedback</div>+ <div id="prev-feedback-text"></div>+ </div>+ </div>+ </div>+ </div>++ <div class="nav" id="outputs-nav">+ <button class="nav-btn" id="prev-btn" onclick="navigate(-1)">← Previous</button>+ <button class="done-btn" id="done-btn" onclick="showDoneDialog()">Submit All Reviews</button>+ <button class="nav-btn" id="next-btn" onclick="navigate(1)">Next →</button>+ </div>+ </div><!-- end panel-outputs -->++ <!-- Benchmark panel (quantitative stats) -->+ <div class="view-panel" id="panel-benchmark">+ <div class="benchmark-view" id="benchmark-content">+ <div class="benchmark-empty">No benchmark data available. Run a benchmark to see quantitative results here.</div>+ </div>+ </div>+ </div>++ <!-- Done overlay -->+ <div class="done-overlay" id="done-overlay">+ <div class="done-card">+ <h2>Review Complete</h2>+ <p>Your feedback has been saved. Go back to your Claude Code session and tell Claude you're done reviewing.</p>+ <div class="btn-row">+ <button onclick="closeDoneDialog()">OK</button>+ </div>+ </div>+ </div>++ <!-- Toast -->+ <div class="toast" id="toast"></div>++ <script>+ // ---- Embedded data (injected by generate_review.py) ----+ /*__EMBEDDED_DATA__*/++ // ---- State ----+ let feedbackMap = {}; // run_id -> feedback text+ let currentIndex = 0;+ let visitedRuns = new Set();++ // ---- Init ----+ async function init() {+ // Load saved feedback from server — but only if this isn't a fresh+ // iteration (indicated by previous_feedback being present). When+ // previous feedback exists, the feedback.json on disk is stale from+ // the prior iteration and should not pre-fill the textareas.+ const hasPrevious = Object.keys(EMBEDDED_DATA.previous_feedback || {}).length > 0+ || Object.keys(EMBEDDED_DATA.previous_outputs || {}).length > 0;+ if (!hasPrevious) {+ try {+ const resp = await fetch("/api/feedback");+ const data = await resp.json();+ if (data.reviews) {+ for (const r of data.reviews) feedbackMap[r.run_id] = r.feedback;+ }+ } catch { /* first run, no feedback yet */ }+ }++ document.getElementById("skill-name").textContent = EMBEDDED_DATA.skill_name;+ showRun(0);++ // Wire up feedback auto-save+ const textarea = document.getElementById("feedback");+ let saveTimeout = null;+ textarea.addEventListener("input", () => {+ clearTimeout(saveTimeout);+ document.getElementById("feedback-status").textContent = "";+ saveTimeout = setTimeout(() => saveCurrentFeedback(), 800);+ });+ }++ // ---- Navigation ----+ function navigate(delta) {+ const newIndex = currentIndex + delta;+ if (newIndex >= 0 && newIndex < EMBEDDED_DATA.runs.length) {+ saveCurrentFeedback();+ showRun(newIndex);+ }+ }++ function updateNavButtons() {+ document.getElementById("prev-btn").disabled = currentIndex === 0;+ document.getElementById("next-btn").disabled =+ currentIndex === EMBEDDED_DATA.runs.length - 1;+ }++ // ---- Show a run ----+ function showRun(index) {+ currentIndex = index;+ const run = EMBEDDED_DATA.runs[index];++ // Progress+ document.getElementById("progress").textContent =+ `${index + 1} of ${EMBEDDED_DATA.runs.length}`;++ // Prompt+ document.getElementById("prompt-text").textContent = run.prompt;++ // Config badge+ const badge = document.getElementById("config-badge");+ const configMatch = run.id.match(/(with_skill|without_skill|new_skill|old_skill)/);+ if (configMatch) {+ const config = configMatch[1];+ const isBaseline = config === "without_skill" || config === "old_skill";+ badge.textContent = config.replace(/_/g, " ");+ badge.className = "config-badge " + (isBaseline ? "config-baseline" : "config-primary");+ badge.style.display = "inline-block";+ } else {+ badge.style.display = "none";+ }++ // Outputs+ renderOutputs(run);++ // Previous outputs+ renderPrevOutputs(run);++ // Grades+ renderGrades(run);++ // Previous feedback+ const prevFb = (EMBEDDED_DATA.previous_feedback || {})[run.id];+ const prevEl = document.getElementById("prev-feedback");+ if (prevFb) {+ document.getElementById("prev-feedback-text").textContent = prevFb;+ prevEl.style.display = "block";+ } else {+ prevEl.style.display = "none";+ }++ // Feedback+ document.getElementById("feedback").value = feedbackMap[run.id] || "";+ document.getElementById("feedback-status").textContent = "";++ updateNavButtons();++ // Track visited runs and promote done button when all visited+ visitedRuns.add(index);+ const doneBtn = document.getElementById("done-btn");+ if (visitedRuns.size >= EMBEDDED_DATA.runs.length) {+ doneBtn.classList.add("ready");+ }++ // Scroll main content to top+ document.querySelector(".main").scrollTop = 0;+ }++ // ---- Render outputs ----+ function renderOutputs(run) {+ const container = document.getElementById("outputs-body");+ container.innerHTML = "";++ const outputs = run.outputs || [];+ if (outputs.length === 0) {+ container.innerHTML = '<div class="empty-state">No output files</div>';+ return;+ }++ for (const file of outputs) {+ const fileDiv = document.createElement("div");+ fileDiv.className = "output-file";++ // Always show file header with download link+ const header = document.createElement("div");+ header.className = "output-file-header";+ const nameSpan = document.createElement("span");+ nameSpan.textContent = file.name;+ header.appendChild(nameSpan);+ const dlBtn = document.createElement("a");+ dlBtn.className = "dl-btn";+ dlBtn.textContent = "Download";+ dlBtn.download = file.name;+ dlBtn.href = getDownloadUri(file);+ header.appendChild(dlBtn);+ fileDiv.appendChild(header);++ const content = document.createElement("div");+ content.className = "output-file-content";++ if (file.type === "text") {+ const pre = document.createElement("pre");+ pre.textContent = file.content;+ content.appendChild(pre);+ } else if (file.type === "image") {+ const img = document.createElement("img");+ img.src = file.data_uri;+ img.alt = file.name;+ content.appendChild(img);+ } else if (file.type === "pdf") {+ const iframe = document.createElement("iframe");+ iframe.src = file.data_uri;+ content.appendChild(iframe);+ } else if (file.type === "xlsx") {+ renderXlsx(content, file.data_b64);+ } else if (file.type === "binary") {+ const a = document.createElement("a");+ a.className = "download-link";+ a.href = file.data_uri;+ a.download = file.name;+ a.textContent = "Download " + file.name;+ content.appendChild(a);+ } else if (file.type === "error") {+ const pre = document.createElement("pre");+ pre.textContent = file.content;+ pre.style.color = "var(--red)";+ content.appendChild(pre);+ }++ fileDiv.appendChild(content);+ container.appendChild(fileDiv);+ }+ }++ // ---- XLSX rendering via SheetJS ----+ function renderXlsx(container, b64Data) {+ try {+ const raw = Uint8Array.from(atob(b64Data), c => c.charCodeAt(0));+ const wb = XLSX.read(raw, { type: "array" });++ for (let i = 0; i < wb.SheetNames.length; i++) {+ const sheetName = wb.SheetNames[i];+ const ws = wb.Sheets[sheetName];++ if (wb.SheetNames.length > 1) {+ const sheetLabel = document.createElement("div");+ sheetLabel.style.cssText =+ "font-weight:600; font-size:0.8rem; color:#b0aea5; margin-top:0.5rem; margin-bottom:0.25rem;";+ sheetLabel.textContent = "Sheet: " + sheetName;+ container.appendChild(sheetLabel);+ }++ const htmlStr = XLSX.utils.sheet_to_html(ws, { editable: false });+ const wrapper = document.createElement("div");+ wrapper.innerHTML = htmlStr;+ container.appendChild(wrapper);+ }+ } catch (err) {+ container.textContent = "Error rendering spreadsheet: " + err.message;+ }+ }++ // ---- Grades ----+ function renderGrades(run) {+ const section = document.getElementById("grades-section");+ const content = document.getElementById("grades-content");++ if (!run.grading) {+ section.style.display = "none";+ return;+ }++ const grading = run.grading;+ section.style.display = "block";+ // Reset to collapsed+ content.classList.remove("open");+ document.getElementById("grades-arrow").classList.remove("open");++ const summary = grading.summary || {};+ const expectations = grading.expectations || [];++ let html = '<div style="padding: 1rem;">';++ // Summary line+ const passRate = summary.pass_rate != null+ ? Math.round(summary.pass_rate * 100) + "%"+ : "?";+ const badgeClass = summary.pass_rate >= 0.8 ? "grade-pass" : summary.pass_rate >= 0.5 ? "" : "grade-fail";+ html += '<div class="grades-summary">';+ html += '<span class="grade-badge ' + badgeClass + '">' + passRate + '</span>';+ html += '<span>' + (summary.passed || 0) + ' passed, ' + (summary.failed || 0) + ' failed of ' + (summary.total || 0) + '</span>';+ html += '</div>';++ // Assertions list+ html += '<ul class="assertion-list">';+ for (const exp of expectations) {+ const statusClass = exp.passed ? "pass" : "fail";+ const statusIcon = exp.passed ? "\u2713" : "\u2717";+ html += '<li class="assertion-item">';+ html += '<span class="assertion-status ' + statusClass + '">' + statusIcon + '</span>';+ html += '<span>' + escapeHtml(exp.text) + '</span>';+ if (exp.evidence) {+ html += '<div class="assertion-evidence">' + escapeHtml(exp.evidence) + '</div>';+ }+ html += '</li>';+ }+ html += '</ul>';++ html += '</div>';+ content.innerHTML = html;+ }++ function toggleGrades() {+ const content = document.getElementById("grades-content");+ const arrow = document.getElementById("grades-arrow");+ content.classList.toggle("open");+ arrow.classList.toggle("open");+ }++ // ---- Previous outputs (collapsible) ----+ function renderPrevOutputs(run) {+ const section = document.getElementById("prev-outputs-section");+ const content = document.getElementById("prev-outputs-content");+ const prevOutputs = (EMBEDDED_DATA.previous_outputs || {})[run.id];++ if (!prevOutputs || prevOutputs.length === 0) {+ section.style.display = "none";+ return;+ }++ section.style.display = "block";+ // Reset to collapsed+ content.classList.remove("open");+ document.getElementById("prev-outputs-arrow").classList.remove("open");++ // Render the files into the content area+ content.innerHTML = "";+ const wrapper = document.createElement("div");+ wrapper.style.padding = "1rem";++ for (const file of prevOutputs) {+ const fileDiv = document.createElement("div");+ fileDiv.className = "output-file";++ const header = document.createElement("div");+ header.className = "output-file-header";+ const nameSpan = document.createElement("span");+ nameSpan.textContent = file.name;+ header.appendChild(nameSpan);+ const dlBtn = document.createElement("a");+ dlBtn.className = "dl-btn";+ dlBtn.textContent = "Download";+ dlBtn.download = file.name;+ dlBtn.href = getDownloadUri(file);+ header.appendChild(dlBtn);+ fileDiv.appendChild(header);++ const fc = document.createElement("div");+ fc.className = "output-file-content";++ if (file.type === "text") {+ const pre = document.createElement("pre");+ pre.textContent = file.content;+ fc.appendChild(pre);+ } else if (file.type === "image") {+ const img = document.createElement("img");+ img.src = file.data_uri;+ img.alt = file.name;+ fc.appendChild(img);+ } else if (file.type === "pdf") {+ const iframe = document.createElement("iframe");+ iframe.src = file.data_uri;+ fc.appendChild(iframe);+ } else if (file.type === "xlsx") {+ renderXlsx(fc, file.data_b64);+ } else if (file.type === "binary") {+ const a = document.createElement("a");+ a.className = "download-link";+ a.href = file.data_uri;+ a.download = file.name;+ a.textContent = "Download " + file.name;+ fc.appendChild(a);+ }++ fileDiv.appendChild(fc);+ wrapper.appendChild(fileDiv);+ }++ content.appendChild(wrapper);+ }++ function togglePrevOutputs() {+ const content = document.getElementById("prev-outputs-content");+ const arrow = document.getElementById("prev-outputs-arrow");+ content.classList.toggle("open");+ arrow.classList.toggle("open");+ }++ // ---- Feedback (saved to server -> feedback.json) ----+ function saveCurrentFeedback() {+ const run = EMBEDDED_DATA.runs[currentIndex];+ const text = document.getElementById("feedback").value;++ if (text.trim() === "") {+ delete feedbackMap[run.id];+ } else {+ feedbackMap[run.id] = text;+ }++ // Build reviews array from map+ const reviews = [];+ for (const [run_id, feedback] of Object.entries(feedbackMap)) {+ if (feedback.trim()) {+ reviews.push({ run_id, feedback, timestamp: new Date().toISOString() });+ }+ }++ fetch("/api/feedback", {+ method: "POST",+ headers: { "Content-Type": "application/json" },+ body: JSON.stringify({ reviews, status: "in_progress" }),+ }).then(() => {+ document.getElementById("feedback-status").textContent = "Saved";+ }).catch(() => {+ // Static mode or server unavailable — no-op on auto-save,+ // feedback will be downloaded on final submit+ document.getElementById("feedback-status").textContent = "Will download on submit";+ });+ }++ // ---- Done ----+ function showDoneDialog() {+ // Save current textarea to feedbackMap (but don't POST yet)+ const run = EMBEDDED_DATA.runs[currentIndex];+ const text = document.getElementById("feedback").value;+ if (text.trim() === "") {+ delete feedbackMap[run.id];+ } else {+ feedbackMap[run.id] = text;+ }++ // POST once with status: complete — include ALL runs so the model+ // can distinguish "no feedback" (looks good) from "not reviewed"+ const reviews = [];+ const ts = new Date().toISOString();+ for (const r of EMBEDDED_DATA.runs) {+ reviews.push({ run_id: r.id, feedback: feedbackMap[r.id] || "", timestamp: ts });+ }+ const payload = JSON.stringify({ reviews, status: "complete" }, null, 2);+ fetch("/api/feedback", {+ method: "POST",+ headers: { "Content-Type": "application/json" },+ body: payload,+ }).then(() => {+ document.getElementById("done-overlay").classList.add("visible");+ }).catch(() => {+ // Server not available (static mode) — download as file+ const blob = new Blob([payload], { type: "application/json" });+ const url = URL.createObjectURL(blob);+ const a = document.createElement("a");+ a.href = url;+ a.download = "feedback.json";+ a.click();+ URL.revokeObjectURL(url);+ document.getElementById("done-overlay").classList.add("visible");+ });+ }++ function closeDoneDialog() {+ // Reset status back to in_progress+ saveCurrentFeedback();+ document.getElementById("done-overlay").classList.remove("visible");+ }++ // ---- Toast ----+ function showToast(message) {+ const toast = document.getElementById("toast");+ toast.textContent = message;+ toast.classList.add("visible");+ setTimeout(() => toast.classList.remove("visible"), 2000);+ }++ // ---- Keyboard nav ----+ document.addEventListener("keydown", (e) => {+ // Don't capture when typing in textarea+ if (e.target.tagName === "TEXTAREA") return;++ if (e.key === "ArrowLeft" || e.key === "ArrowUp") {+ e.preventDefault();+ navigate(-1);+ } else if (e.key === "ArrowRight" || e.key === "ArrowDown") {+ e.preventDefault();+ navigate(1);+ }+ });++ // ---- Util ----+ function getDownloadUri(file) {+ if (file.data_uri) return file.data_uri;+ if (file.data_b64) return "data:application/octet-stream;base64," + file.data_b64;+ if (file.type === "text") return "data:text/plain;charset=utf-8," + encodeURIComponent(file.content);+ return "#";+ }++ function escapeHtml(text) {+ const div = document.createElement("div");+ div.textContent = text;+ return div.innerHTML;+ }++ // ---- View switching ----+ function switchView(view) {+ document.querySelectorAll(".view-tab").forEach(t => t.classList.remove("active"));+ document.querySelectorAll(".view-panel").forEach(p => p.classList.remove("active"));+ document.querySelector(`[onclick="switchView('${view}')"]`).classList.add("active");+ document.getElementById("panel-" + view).classList.add("active");+ }++ // ---- Benchmark rendering ----+ function renderBenchmark() {+ const data = EMBEDDED_DATA.benchmark;+ if (!data) return;++ // Show the tabs+ document.getElementById("view-tabs").style.display = "flex";++ const container = document.getElementById("benchmark-content");+ const summary = data.run_summary || {};+ const metadata = data.metadata || {};+ const notes = data.notes || [];++ let html = "";++ // Header+ html += "<h2 style='font-family: Poppins, sans-serif; margin-bottom: 0.5rem;'>Benchmark Results</h2>";+ html += "<p style='color: var(--text-muted); font-size: 0.875rem; margin-bottom: 1.25rem;'>";+ if (metadata.skill_name) html += "<strong>" + escapeHtml(metadata.skill_name) + "</strong> — ";+ if (metadata.timestamp) html += metadata.timestamp + " — ";+ if (metadata.evals_run) html += "Evals: " + metadata.evals_run.join(", ") + " — ";+ html += (metadata.runs_per_configuration || "?") + " runs per configuration";+ html += "</p>";++ // Summary table+ html += '<table class="benchmark-table">';++ function fmtStat(stat, pct) {+ if (!stat) return "—";+ const suffix = pct ? "%" : "";+ const m = pct ? (stat.mean * 100).toFixed(0) : stat.mean.toFixed(1);+ const s = pct ? (stat.stddev * 100).toFixed(0) : stat.stddev.toFixed(1);+ return m + suffix + " ± " + s + suffix;+ }++ function deltaClass(val) {+ if (!val) return "";+ const n = parseFloat(val);+ if (n > 0) return "benchmark-delta-positive";+ if (n < 0) return "benchmark-delta-negative";+ return "";+ }++ // Discover config names dynamically (everything except "delta")+ const configs = Object.keys(summary).filter(k => k !== "delta");+ const configA = configs[0] || "config_a";+ const configB = configs[1] || "config_b";+ const labelA = configA.replace(/_/g, " ").replace(/\b\w/g, c => c.toUpperCase());+ const labelB = configB.replace(/_/g, " ").replace(/\b\w/g, c => c.toUpperCase());+ const a = summary[configA] || {};+ const b = summary[configB] || {};+ const delta = summary.delta || {};++ html += "<thead><tr><th>Metric</th><th>" + escapeHtml(labelA) + "</th><th>" + escapeHtml(labelB) + "</th><th>Delta</th></tr></thead>";+ html += "<tbody>";++ html += "<tr><td><strong>Pass Rate</strong></td>";+ html += "<td>" + fmtStat(a.pass_rate, true) + "</td>";+ html += "<td>" + fmtStat(b.pass_rate, true) + "</td>";+ html += '<td class="' + deltaClass(delta.pass_rate) + '">' + (delta.pass_rate || "—") + "</td></tr>";++ // Time (only show row if data exists)+ if (a.time_seconds || b.time_seconds) {+ html += "<tr><td><strong>Time (s)</strong></td>";+ html += "<td>" + fmtStat(a.time_seconds, false) + "</td>";+ html += "<td>" + fmtStat(b.time_seconds, false) + "</td>";+ html += '<td class="' + deltaClass(delta.time_seconds) + '">' + (delta.time_seconds ? delta.time_seconds + "s" : "—") + "</td></tr>";+ }++ // Tokens (only show row if data exists)+ if (a.tokens || b.tokens) {+ html += "<tr><td><strong>Tokens</strong></td>";+ html += "<td>" + fmtStat(a.tokens, false) + "</td>";+ html += "<td>" + fmtStat(b.tokens, false) + "</td>";+ html += '<td class="' + deltaClass(delta.tokens) + '">' + (delta.tokens || "—") + "</td></tr>";+ }++ html += "</tbody></table>";++ // Per-eval breakdown (if runs data available)+ const runs = data.runs || [];+ if (runs.length > 0) {+ const evalIds = [...new Set(runs.map(r => r.eval_id))].sort((a, b) => a - b);++ html += "<h3 style='font-family: Poppins, sans-serif; margin-bottom: 0.75rem;'>Per-Eval Breakdown</h3>";++ const hasTime = runs.some(r => r.result && r.result.time_seconds != null);+ const hasErrors = runs.some(r => r.result && r.result.errors > 0);++ for (const evalId of evalIds) {+ const evalRuns = runs.filter(r => r.eval_id === evalId);+ const evalName = evalRuns[0] && evalRuns[0].eval_name ? evalRuns[0].eval_name : "Eval " + evalId;++ html += "<h4 style='font-family: Poppins, sans-serif; margin: 1rem 0 0.5rem; color: var(--text);'>" + escapeHtml(evalName) + "</h4>";+ html += '<table class="benchmark-table">';+ html += "<thead><tr><th>Config</th><th>Run</th><th>Pass Rate</th>";+ if (hasTime) html += "<th>Time (s)</th>";+ if (hasErrors) html += "<th>Crashes During Execution</th>";+ html += "</tr></thead>";+ html += "<tbody>";++ // Group by config and render with average rows+ const configGroups = [...new Set(evalRuns.map(r => r.configuration))];+ for (let ci = 0; ci < configGroups.length; ci++) {+ const config = configGroups[ci];+ const configRuns = evalRuns.filter(r => r.configuration === config);+ if (configRuns.length === 0) continue;++ const rowClass = ci === 0 ? "benchmark-row-with" : "benchmark-row-without";+ const configLabel = config.replace(/_/g, " ").replace(/\b\w/g, c => c.toUpperCase());++ for (const run of configRuns) {+ const r = run.result || {};+ const prClass = r.pass_rate >= 0.8 ? "benchmark-delta-positive" : r.pass_rate < 0.5 ? "benchmark-delta-negative" : "";+ html += '<tr class="' + rowClass + '">';+ html += "<td>" + configLabel + "</td>";+ html += "<td>" + run.run_number + "</td>";+ html += '<td class="' + prClass + '">' + ((r.pass_rate || 0) * 100).toFixed(0) + "% (" + (r.passed || 0) + "/" + (r.total || 0) + ")</td>";+ if (hasTime) html += "<td>" + (r.time_seconds != null ? r.time_seconds.toFixed(1) : "—") + "</td>";+ if (hasErrors) html += "<td>" + (r.errors || 0) + "</td>";+ html += "</tr>";+ }++ // Average row+ const rates = configRuns.map(r => (r.result || {}).pass_rate || 0);+ const avgRate = rates.reduce((a, b) => a + b, 0) / rates.length;+ const avgPrClass = avgRate >= 0.8 ? "benchmark-delta-positive" : avgRate < 0.5 ? "benchmark-delta-negative" : "";+ html += '<tr class="benchmark-row-avg ' + rowClass + '">';+ html += "<td>" + configLabel + "</td>";+ html += "<td>Avg</td>";+ html += '<td class="' + avgPrClass + '">' + (avgRate * 100).toFixed(0) + "%</td>";+ if (hasTime) {+ const times = configRuns.map(r => (r.result || {}).time_seconds).filter(t => t != null);+ html += "<td>" + (times.length ? (times.reduce((a, b) => a + b, 0) / times.length).toFixed(1) : "—") + "</td>";+ }+ if (hasErrors) html += "<td></td>";+ html += "</tr>";+ }+ html += "</tbody></table>";++ // Per-assertion detail for this eval+ const runsWithExpectations = {};+ for (const config of configGroups) {+ runsWithExpectations[config] = evalRuns.filter(r => r.configuration === config && r.expectations && r.expectations.length > 0);+ }+ const hasAnyExpectations = Object.values(runsWithExpectations).some(runs => runs.length > 0);+ if (hasAnyExpectations) {+ // Collect all unique assertion texts across all configs+ const allAssertions = [];+ const seen = new Set();+ for (const config of configGroups) {+ for (const run of runsWithExpectations[config]) {+ for (const exp of (run.expectations || [])) {+ if (!seen.has(exp.text)) {+ seen.add(exp.text);+ allAssertions.push(exp.text);+ }+ }+ }+ }++ html += '<table class="benchmark-table" style="margin-top: 0.5rem;">';+ html += "<thead><tr><th>Assertion</th>";+ for (const config of configGroups) {+ const label = config.replace(/_/g, " ").replace(/\b\w/g, c => c.toUpperCase());+ html += "<th>" + escapeHtml(label) + "</th>";+ }+ html += "</tr></thead><tbody>";++ for (const assertionText of allAssertions) {+ html += "<tr><td>" + escapeHtml(assertionText) + "</td>";++ for (const config of configGroups) {+ html += "<td>";+ for (const run of runsWithExpectations[config]) {+ const exp = (run.expectations || []).find(e => e.text === assertionText);+ if (exp) {+ const cls = exp.passed ? "benchmark-delta-positive" : "benchmark-delta-negative";+ const icon = exp.passed ? "\u2713" : "\u2717";+ html += '<span class="' + cls + '" title="Run ' + run.run_number + ': ' + escapeHtml(exp.evidence || "") + '">' + icon + "</span> ";+ } else {+ html += "— ";+ }+ }+ html += "</td>";+ }+ html += "</tr>";+ }+ html += "</tbody></table>";+ }+ }+ }++ // Notes+ if (notes.length > 0) {+ html += '<div class="benchmark-notes">';+ html += "<h3>Analysis Notes</h3>";+ html += "<ul>";+ for (const note of notes) {+ html += "<li>" + escapeHtml(note) + "</li>";+ }+ html += "</ul></div>";+ }++ container.innerHTML = html;+ }++ // ---- Start ----+ init();+ renderBenchmark();+ </script>+</body>+</html>diff --git a/.opencode/skills/skill-creator/references/schemas.md b/.opencode/skills/skill-creator/references/schemas.mdnew file mode 100644index 0000000..b6eeaa2--- /dev/null+++ b/.opencode/skills/skill-creator/references/schemas.md@@ -0,0 +1,430 @@+# JSON Schemas++This document defines the JSON schemas used by skill-creator.++---++## evals.json++Defines the evals for a skill. Located at `evals/evals.json` within the skill directory.++```json+{+ "skill_name": "example-skill",+ "evals": [+ {+ "id": 1,+ "prompt": "User's example prompt",+ "expected_output": "Description of expected result",+ "files": ["evals/files/sample1.pdf"],+ "expectations": [+ "The output includes X",+ "The skill used script Y"+ ]+ }+ ]+}+```++**Fields:**+- `skill_name`: Name matching the skill's frontmatter+- `evals[].id`: Unique integer identifier+- `evals[].prompt`: The task to execute+- `evals[].expected_output`: Human-readable description of success+- `evals[].files`: Optional list of input file paths (relative to skill root)+- `evals[].expectations`: List of verifiable statements++---++## history.json++Tracks version progression in Improve mode. Located at workspace root.++```json+{+ "started_at": "2026-01-15T10:30:00Z",+ "skill_name": "pdf",+ "current_best": "v2",+ "iterations": [+ {+ "version": "v0",+ "parent": null,+ "expectation_pass_rate": 0.65,+ "grading_result": "baseline",+ "is_current_best": false+ },+ {+ "version": "v1",+ "parent": "v0",+ "expectation_pass_rate": 0.75,+ "grading_result": "won",+ "is_current_best": false+ },+ {+ "version": "v2",+ "parent": "v1",+ "expectation_pass_rate": 0.85,+ "grading_result": "won",+ "is_current_best": true+ }+ ]+}+```++**Fields:**+- `started_at`: ISO timestamp of when improvement started+- `skill_name`: Name of the skill being improved+- `current_best`: Version identifier of the best performer+- `iterations[].version`: Version identifier (v0, v1, ...)+- `iterations[].parent`: Parent version this was derived from+- `iterations[].expectation_pass_rate`: Pass rate from grading+- `iterations[].grading_result`: "baseline", "won", "lost", or "tie"+- `iterations[].is_current_best`: Whether this is the current best version++---++## grading.json++Output from the grader agent. Located at `<run-dir>/grading.json`.++```json+{+ "expectations": [+ {+ "text": "The output includes the name 'John Smith'",+ "passed": true,+ "evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'"+ },+ {+ "text": "The spreadsheet has a SUM formula in cell B10",+ "passed": false,+ "evidence": "No spreadsheet was created. The output was a text file."+ }+ ],+ "summary": {+ "passed": 2,+ "failed": 1,+ "total": 3,+ "pass_rate": 0.67+ },+ "execution_metrics": {+ "tool_calls": {+ "Read": 5,+ "Write": 2,+ "Bash": 8+ },+ "total_tool_calls": 15,+ "total_steps": 6,+ "errors_encountered": 0,+ "output_chars": 12450,+ "transcript_chars": 3200+ },+ "timing": {+ "executor_duration_seconds": 165.0,+ "grader_duration_seconds": 26.0,+ "total_duration_seconds": 191.0+ },+ "claims": [+ {+ "claim": "The form has 12 fillable fields",+ "type": "factual",+ "verified": true,+ "evidence": "Counted 12 fields in field_info.json"+ }+ ],+ "user_notes_summary": {+ "uncertainties": ["Used 2023 data, may be stale"],+ "needs_review": [],+ "workarounds": ["Fell back to text overlay for non-fillable fields"]+ },+ "eval_feedback": {+ "suggestions": [+ {+ "assertion": "The output includes the name 'John Smith'",+ "reason": "A hallucinated document that mentions the name would also pass"+ }+ ],+ "overall": "Assertions check presence but not correctness."+ }+}+```++**Fields:**+- `expectations[]`: Graded expectations with evidence+- `summary`: Aggregate pass/fail counts+- `execution_metrics`: Tool usage and output size (from executor's metrics.json)+- `timing`: Wall clock timing (from timing.json)+- `claims`: Extracted and verified claims from the output+- `user_notes_summary`: Issues flagged by the executor+- `eval_feedback`: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising++---++## metrics.json++Output from the executor agent. Located at `<run-dir>/outputs/metrics.json`.++```json+{+ "tool_calls": {+ "Read": 5,+ "Write": 2,+ "Bash": 8,+ "Edit": 1,+ "Glob": 2,+ "Grep": 0+ },+ "total_tool_calls": 18,+ "total_steps": 6,+ "files_created": ["filled_form.pdf", "field_values.json"],+ "errors_encountered": 0,+ "output_chars": 12450,+ "transcript_chars": 3200+}+```++**Fields:**+- `tool_calls`: Count per tool type+- `total_tool_calls`: Sum of all tool calls+- `total_steps`: Number of major execution steps+- `files_created`: List of output files created+- `errors_encountered`: Number of errors during execution+- `output_chars`: Total character count of output files+- `transcript_chars`: Character count of transcript++---++## timing.json++Wall clock timing for a run. Located at `<run-dir>/timing.json`.++**How to capture:** When a subagent task completes, the task notification includes `total_tokens` and `duration_ms`. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.++```json+{+ "total_tokens": 84852,+ "duration_ms": 23332,+ "total_duration_seconds": 23.3,+ "executor_start": "2026-01-15T10:30:00Z",+ "executor_end": "2026-01-15T10:32:45Z",+ "executor_duration_seconds": 165.0,+ "grader_start": "2026-01-15T10:32:46Z",+ "grader_end": "2026-01-15T10:33:12Z",+ "grader_duration_seconds": 26.0+}+```++---++## benchmark.json++Output from Benchmark mode. Located at `benchmarks/<timestamp>/benchmark.json`.++```json+{+ "metadata": {+ "skill_name": "pdf",+ "skill_path": "/path/to/pdf",+ "executor_model": "claude-sonnet-4-20250514",+ "analyzer_model": "most-capable-model",+ "timestamp": "2026-01-15T10:30:00Z",+ "evals_run": [1, 2, 3],+ "runs_per_configuration": 3+ },++ "runs": [+ {+ "eval_id": 1,+ "eval_name": "Ocean",+ "configuration": "with_skill",+ "run_number": 1,+ "result": {+ "pass_rate": 0.85,+ "passed": 6,+ "failed": 1,+ "total": 7,+ "time_seconds": 42.5,+ "tokens": 3800,+ "tool_calls": 18,+ "errors": 0+ },+ "expectations": [+ {"text": "...", "passed": true, "evidence": "..."}+ ],+ "notes": [+ "Used 2023 data, may be stale",+ "Fell back to text overlay for non-fillable fields"+ ]+ }+ ],++ "run_summary": {+ "with_skill": {+ "pass_rate": {"mean": 0.85, "stddev": 0.05, "min": 0.80, "max": 0.90},+ "time_seconds": {"mean": 45.0, "stddev": 12.0, "min": 32.0, "max": 58.0},+ "tokens": {"mean": 3800, "stddev": 400, "min": 3200, "max": 4100}+ },+ "without_skill": {+ "pass_rate": {"mean": 0.35, "stddev": 0.08, "min": 0.28, "max": 0.45},+ "time_seconds": {"mean": 32.0, "stddev": 8.0, "min": 24.0, "max": 42.0},+ "tokens": {"mean": 2100, "stddev": 300, "min": 1800, "max": 2500}+ },+ "delta": {+ "pass_rate": "+0.50",+ "time_seconds": "+13.0",+ "tokens": "+1700"+ }+ },++ "notes": [+ "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value",+ "Eval 3 shows high variance (50% ± 40%) - may be flaky or model-dependent",+ "Without-skill runs consistently fail on table extraction expectations",+ "Skill adds 13s average execution time but improves pass rate by 50%"+ ]+}+```++**Fields:**+- `metadata`: Information about the benchmark run+ - `skill_name`: Name of the skill+ - `timestamp`: When the benchmark was run+ - `evals_run`: List of eval names or IDs+ - `runs_per_configuration`: Number of runs per config (e.g. 3)+- `runs[]`: Individual run results+ - `eval_id`: Numeric eval identifier+ - `eval_name`: Human-readable eval name (used as section header in the viewer)+ - `configuration`: Must be `"with_skill"` or `"without_skill"` (the viewer uses this exact string for grouping and color coding)+ - `run_number`: Integer run number (1, 2, 3...)+ - `result`: Nested object with `pass_rate`, `passed`, `total`, `time_seconds`, `tokens`, `errors`+- `run_summary`: Statistical aggregates per configuration+ - `with_skill` / `without_skill`: Each contains `pass_rate`, `time_seconds`, `tokens` objects with `mean` and `stddev` fields+ - `delta`: Difference strings like `"+0.50"`, `"+13.0"`, `"+1700"`+- `notes`: Freeform observations from the analyzer++**Important:** The viewer reads these field names exactly. Using `config` instead of `configuration`, or putting `pass_rate` at the top level of a run instead of nested under `result`, will cause the viewer to show empty/zero values. Always reference this schema when generating benchmark.json manually.++---++## comparison.json++Output from blind comparator. Located at `<grading-dir>/comparison-N.json`.++```json+{+ "winner": "A",+ "reasoning": "Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.",+ "rubric": {+ "A": {+ "content": {+ "correctness": 5,+ "completeness": 5,+ "accuracy": 4+ },+ "structure": {+ "organization": 4,+ "formatting": 5,+ "usability": 4+ },+ "content_score": 4.7,+ "structure_score": 4.3,+ "overall_score": 9.0+ },+ "B": {+ "content": {+ "correctness": 3,+ "completeness": 2,+ "accuracy": 3+ },+ "structure": {+ "organization": 3,+ "formatting": 2,+ "usability": 3+ },+ "content_score": 2.7,+ "structure_score": 2.7,+ "overall_score": 5.4+ }+ },+ "output_quality": {+ "A": {+ "score": 9,+ "strengths": ["Complete solution", "Well-formatted", "All fields present"],+ "weaknesses": ["Minor style inconsistency in header"]+ },+ "B": {+ "score": 5,+ "strengths": ["Readable output", "Correct basic structure"],+ "weaknesses": ["Missing date field", "Formatting inconsistencies", "Partial data extraction"]+ }+ },+ "expectation_results": {+ "A": {+ "passed": 4,+ "total": 5,+ "pass_rate": 0.80,+ "details": [+ {"text": "Output includes name", "passed": true}+ ]+ },+ "B": {+ "passed": 3,+ "total": 5,+ "pass_rate": 0.60,+ "details": [+ {"text": "Output includes name", "passed": true}+ ]+ }+ }+}+```++---++## analysis.json++Output from post-hoc analyzer. Located at `<grading-dir>/analysis.json`.++```json+{+ "comparison_summary": {+ "winner": "A",+ "winner_skill": "path/to/winner/skill",+ "loser_skill": "path/to/loser/skill",+ "comparator_reasoning": "Brief summary of why comparator chose winner"+ },+ "winner_strengths": [+ "Clear step-by-step instructions for handling multi-page documents",+ "Included validation script that caught formatting errors"+ ],+ "loser_weaknesses": [+ "Vague instruction 'process the document appropriately' led to inconsistent behavior",+ "No script for validation, agent had to improvise"+ ],+ "instruction_following": {+ "winner": {+ "score": 9,+ "issues": ["Minor: skipped optional logging step"]+ },+ "loser": {+ "score": 6,+ "issues": [+ "Did not use the skill's formatting template",+ "Invented own approach instead of following step 3"+ ]+ }+ },+ "improvement_suggestions": [+ {+ "priority": "high",+ "category": "instructions",+ "suggestion": "Replace 'process the document appropriately' with explicit steps",+ "expected_impact": "Would eliminate ambiguity that caused inconsistent behavior"+ }+ ],+ "transcript_insights": {+ "winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script",+ "loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods"+ }+}+```diff --git a/.opencode/skills/skill-creator/scripts/__init__.py b/.opencode/skills/skill-creator/scripts/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/.opencode/skills/skill-creator/scripts/aggregate_benchmark.py b/.opencode/skills/skill-creator/scripts/aggregate_benchmark.pynew file mode 100644index 0000000..3e66e8c--- /dev/null+++ b/.opencode/skills/skill-creator/scripts/aggregate_benchmark.py@@ -0,0 +1,401 @@+#!/usr/bin/env python3+"""+Aggregate individual run results into benchmark summary statistics.++Reads grading.json files from run directories and produces:+- run_summary with mean, stddev, min, max for each metric+- delta between with_skill and without_skill configurations++Usage:+ python aggregate_benchmark.py <benchmark_dir>++Example:+ python aggregate_benchmark.py benchmarks/2026-01-15T10-30-00/++The script supports two directory layouts:++ Workspace layout (from skill-creator iterations):+ <benchmark_dir>/+ └── eval-N/+ ├── with_skill/+ │ ├── run-1/grading.json+ │ └── run-2/grading.json+ └── without_skill/+ ├── run-1/grading.json+ └── run-2/grading.json++ Legacy layout (with runs/ subdirectory):+ <benchmark_dir>/+ └── runs/+ └── eval-N/+ ├── with_skill/+ │ └── run-1/grading.json+ └── without_skill/+ └── run-1/grading.json+"""++import argparse+import json+import math+import sys+from datetime import datetime, timezone+from pathlib import Path+++def calculate_stats(values: list[float]) -> dict:+ """Calculate mean, stddev, min, max for a list of values."""+ if not values:+ return {"mean": 0.0, "stddev": 0.0, "min": 0.0, "max": 0.0}++ n = len(values)+ mean = sum(values) / n++ if n > 1:+ variance = sum((x - mean) ** 2 for x in values) / (n - 1)+ stddev = math.sqrt(variance)+ else:+ stddev = 0.0++ return {+ "mean": round(mean, 4),+ "stddev": round(stddev, 4),+ "min": round(min(values), 4),+ "max": round(max(values), 4)+ }+++def load_run_results(benchmark_dir: Path) -> dict:+ """+ Load all run results from a benchmark directory.++ Returns dict keyed by config name (e.g. "with_skill"/"without_skill",+ or "new_skill"/"old_skill"), each containing a list of run results.+ """+ # Support both layouts: eval dirs directly under benchmark_dir, or under runs/+ runs_dir = benchmark_dir / "runs"+ if runs_dir.exists():+ search_dir = runs_dir+ elif list(benchmark_dir.glob("eval-*")):+ search_dir = benchmark_dir+ else:+ print(f"No eval directories found in {benchmark_dir} or {benchmark_dir / 'runs'}")+ return {}++ results: dict[str, list] = {}++ for eval_idx, eval_dir in enumerate(sorted(search_dir.glob("eval-*"))):+ metadata_path = eval_dir / "eval_metadata.json"+ if metadata_path.exists():+ try:+ with open(metadata_path) as mf:+ eval_id = json.load(mf).get("eval_id", eval_idx)+ except (json.JSONDecodeError, OSError):+ eval_id = eval_idx+ else:+ try:+ eval_id = int(eval_dir.name.split("-")[1])+ except ValueError:+ eval_id = eval_idx++ # Discover config directories dynamically rather than hardcoding names+ for config_dir in sorted(eval_dir.iterdir()):+ if not config_dir.is_dir():+ continue+ # Skip non-config directories (inputs, outputs, etc.)+ if not list(config_dir.glob("run-*")):+ continue+ config = config_dir.name+ if config not in results:+ results[config] = []++ for run_dir in sorted(config_dir.glob("run-*")):+ run_number = int(run_dir.name.split("-")[1])+ grading_file = run_dir / "grading.json"++ if not grading_file.exists():+ print(f"Warning: grading.json not found in {run_dir}")+ continue++ try:+ with open(grading_file) as f:+ grading = json.load(f)+ except json.JSONDecodeError as e:+ print(f"Warning: Invalid JSON in {grading_file}: {e}")+ continue++ # Extract metrics+ result = {+ "eval_id": eval_id,+ "run_number": run_number,+ "pass_rate": grading.get("summary", {}).get("pass_rate", 0.0),+ "passed": grading.get("summary", {}).get("passed", 0),+ "failed": grading.get("summary", {}).get("failed", 0),+ "total": grading.get("summary", {}).get("total", 0),+ }++ # Extract timing — check grading.json first, then sibling timing.json+ timing = grading.get("timing", {})+ result["time_seconds"] = timing.get("total_duration_seconds", 0.0)+ timing_file = run_dir / "timing.json"+ if result["time_seconds"] == 0.0 and timing_file.exists():+ try:+ with open(timing_file) as tf:+ timing_data = json.load(tf)+ result["time_seconds"] = timing_data.get("total_duration_seconds", 0.0)+ result["tokens"] = timing_data.get("total_tokens", 0)+ except json.JSONDecodeError:+ pass++ # Extract metrics if available+ metrics = grading.get("execution_metrics", {})+ result["tool_calls"] = metrics.get("total_tool_calls", 0)+ if not result.get("tokens"):+ result["tokens"] = metrics.get("output_chars", 0)+ result["errors"] = metrics.get("errors_encountered", 0)++ # Extract expectations — viewer requires fields: text, passed, evidence+ raw_expectations = grading.get("expectations", [])+ for exp in raw_expectations:+ if "text" not in exp or "passed" not in exp:+ print(f"Warning: expectation in {grading_file} missing required fields (text, passed, evidence): {exp}")+ result["expectations"] = raw_expectations++ # Extract notes from user_notes_summary+ notes_summary = grading.get("user_notes_summary", {})+ notes = []+ notes.extend(notes_summary.get("uncertainties", []))+ notes.extend(notes_summary.get("needs_review", []))+ notes.extend(notes_summary.get("workarounds", []))+ result["notes"] = notes++ results[config].append(result)++ return results+++def aggregate_results(results: dict) -> dict:+ """+ Aggregate run results into summary statistics.++ Returns run_summary with stats for each configuration and delta.+ """+ run_summary = {}+ configs = list(results.keys())++ for config in configs:+ runs = results.get(config, [])++ if not runs:+ run_summary[config] = {+ "pass_rate": {"mean": 0.0, "stddev": 0.0, "min": 0.0, "max": 0.0},+ "time_seconds": {"mean": 0.0, "stddev": 0.0, "min": 0.0, "max": 0.0},+ "tokens": {"mean": 0, "stddev": 0, "min": 0, "max": 0}+ }+ continue++ pass_rates = [r["pass_rate"] for r in runs]+ times = [r["time_seconds"] for r in runs]+ tokens = [r.get("tokens", 0) for r in runs]++ run_summary[config] = {+ "pass_rate": calculate_stats(pass_rates),+ "time_seconds": calculate_stats(times),+ "tokens": calculate_stats(tokens)+ }++ # Calculate delta between the first two configs (if two exist)+ if len(configs) >= 2:+ primary = run_summary.get(configs[0], {})+ baseline = run_summary.get(configs[1], {})+ else:+ primary = run_summary.get(configs[0], {}) if configs else {}+ baseline = {}++ delta_pass_rate = primary.get("pass_rate", {}).get("mean", 0) - baseline.get("pass_rate", {}).get("mean", 0)+ delta_time = primary.get("time_seconds", {}).get("mean", 0) - baseline.get("time_seconds", {}).get("mean", 0)+ delta_tokens = primary.get("tokens", {}).get("mean", 0) - baseline.get("tokens", {}).get("mean", 0)++ run_summary["delta"] = {+ "pass_rate": f"{delta_pass_rate:+.2f}",+ "time_seconds": f"{delta_time:+.1f}",+ "tokens": f"{delta_tokens:+.0f}"+ }++ return run_summary+++def generate_benchmark(benchmark_dir: Path, skill_name: str = "", skill_path: str = "") -> dict:+ """+ Generate complete benchmark.json from run results.+ """+ results = load_run_results(benchmark_dir)+ run_summary = aggregate_results(results)++ # Build runs array for benchmark.json+ runs = []+ for config in results:+ for result in results[config]:+ runs.append({+ "eval_id": result["eval_id"],+ "configuration": config,+ "run_number": result["run_number"],+ "result": {+ "pass_rate": result["pass_rate"],+ "passed": result["passed"],+ "failed": result["failed"],+ "total": result["total"],+ "time_seconds": result["time_seconds"],+ "tokens": result.get("tokens", 0),+ "tool_calls": result.get("tool_calls", 0),+ "errors": result.get("errors", 0)+ },+ "expectations": result["expectations"],+ "notes": result["notes"]+ })++ # Determine eval IDs from results+ eval_ids = sorted(set(+ r["eval_id"]+ for config in results.values()+ for r in config+ ))++ benchmark = {+ "metadata": {+ "skill_name": skill_name or "<skill-name>",+ "skill_path": skill_path or "<path/to/skill>",+ "executor_model": "<model-name>",+ "analyzer_model": "<model-name>",+ "timestamp": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),+ "evals_run": eval_ids,+ "runs_per_configuration": 3+ },+ "runs": runs,+ "run_summary": run_summary,+ "notes": [] # To be filled by analyzer+ }++ return benchmark+++def generate_markdown(benchmark: dict) -> str:+ """Generate human-readable benchmark.md from benchmark data."""+ metadata = benchmark["metadata"]+ run_summary = benchmark["run_summary"]++ # Determine config names (excluding "delta")+ configs = [k for k in run_summary if k != "delta"]+ config_a = configs[0] if len(configs) >= 1 else "config_a"+ config_b = configs[1] if len(configs) >= 2 else "config_b"+ label_a = config_a.replace("_", " ").title()+ label_b = config_b.replace("_", " ").title()++ lines = [+ f"# Skill Benchmark: {metadata['skill_name']}",+ "",+ f"**Model**: {metadata['executor_model']}",+ f"**Date**: {metadata['timestamp']}",+ f"**Evals**: {', '.join(map(str, metadata['evals_run']))} ({metadata['runs_per_configuration']} runs each per configuration)",+ "",+ "## Summary",+ "",+ f"| Metric | {label_a} | {label_b} | Delta |",+ "|--------|------------|---------------|-------|",+ ]++ a_summary = run_summary.get(config_a, {})+ b_summary = run_summary.get(config_b, {})+ delta = run_summary.get("delta", {})++ # Format pass rate+ a_pr = a_summary.get("pass_rate", {})+ b_pr = b_summary.get("pass_rate", {})+ lines.append(f"| Pass Rate | {a_pr.get('mean', 0)*100:.0f}% ± {a_pr.get('stddev', 0)*100:.0f}% | {b_pr.get('mean', 0)*100:.0f}% ± {b_pr.get('stddev', 0)*100:.0f}% | {delta.get('pass_rate', '—')} |")++ # Format time+ a_time = a_summary.get("time_seconds", {})+ b_time = b_summary.get("time_seconds", {})+ lines.append(f"| Time | {a_time.get('mean', 0):.1f}s ± {a_time.get('stddev', 0):.1f}s | {b_time.get('mean', 0):.1f}s ± {b_time.get('stddev', 0):.1f}s | {delta.get('time_seconds', '—')}s |")++ # Format tokens+ a_tokens = a_summary.get("tokens", {})+ b_tokens = b_summary.get("tokens", {})+ lines.append(f"| Tokens | {a_tokens.get('mean', 0):.0f} ± {a_tokens.get('stddev', 0):.0f} | {b_tokens.get('mean', 0):.0f} ± {b_tokens.get('stddev', 0):.0f} | {delta.get('tokens', '—')} |")++ # Notes section+ if benchmark.get("notes"):+ lines.extend([+ "",+ "## Notes",+ ""+ ])+ for note in benchmark["notes"]:+ lines.append(f"- {note}")++ return "\n".join(lines)+++def main():+ parser = argparse.ArgumentParser(+ description="Aggregate benchmark run results into summary statistics"+ )+ parser.add_argument(+ "benchmark_dir",+ type=Path,+ help="Path to the benchmark directory"+ )+ parser.add_argument(+ "--skill-name",+ default="",+ help="Name of the skill being benchmarked"+ )+ parser.add_argument(+ "--skill-path",+ default="",+ help="Path to the skill being benchmarked"+ )+ parser.add_argument(+ "--output", "-o",+ type=Path,+ help="Output path for benchmark.json (default: <benchmark_dir>/benchmark.json)"+ )++ args = parser.parse_args()++ if not args.benchmark_dir.exists():+ print(f"Directory not found: {args.benchmark_dir}")+ sys.exit(1)++ # Generate benchmark+ benchmark = generate_benchmark(args.benchmark_dir, args.skill_name, args.skill_path)++ # Determine output paths+ output_json = args.output or (args.benchmark_dir / "benchmark.json")+ output_md = output_json.with_suffix(".md")++ # Write benchmark.json+ with open(output_json, "w") as f:+ json.dump(benchmark, f, indent=2)+ print(f"Generated: {output_json}")++ # Write benchmark.md+ markdown = generate_markdown(benchmark)+ with open(output_md, "w") as f:+ f.write(markdown)+ print(f"Generated: {output_md}")++ # Print summary+ run_summary = benchmark["run_summary"]+ configs = [k for k in run_summary if k != "delta"]+ delta = run_summary.get("delta", {})++ print(f"\nSummary:")+ for config in configs:+ pr = run_summary[config]["pass_rate"]["mean"]+ label = config.replace("_", " ").title()+ print(f" {label}: {pr*100:.1f}% pass rate")+ print(f" Delta: {delta.get('pass_rate', '—')}")+++if __name__ == "__main__":+ main()diff --git a/.opencode/skills/skill-creator/scripts/generate_report.py b/.opencode/skills/skill-creator/scripts/generate_report.pynew file mode 100644index 0000000..959e30a--- /dev/null+++ b/.opencode/skills/skill-creator/scripts/generate_report.py@@ -0,0 +1,326 @@+#!/usr/bin/env python3+"""Generate an HTML report from run_loop.py output.++Takes the JSON output from run_loop.py and generates a visual HTML report+showing each description attempt with check/x for each test case.+Distinguishes between train and test queries.+"""++import argparse+import html+import json+import sys+from pathlib import Path+++def generate_html(data: dict, auto_refresh: bool = False, skill_name: str = "") -> str:+ """Generate HTML report from loop output data. If auto_refresh is True, adds a meta refresh tag."""+ history = data.get("history", [])+ holdout = data.get("holdout", 0)+ title_prefix = html.escape(skill_name + " \u2014 ") if skill_name else ""++ # Get all unique queries from train and test sets, with should_trigger info+ train_queries: list[dict] = []+ test_queries: list[dict] = []+ if history:+ for r in history[0].get("train_results", history[0].get("results", [])):+ train_queries.append({"query": r["query"], "should_trigger": r.get("should_trigger", True)})+ if history[0].get("test_results"):+ for r in history[0].get("test_results", []):+ test_queries.append({"query": r["query"], "should_trigger": r.get("should_trigger", True)})++ refresh_tag = ' <meta http-equiv="refresh" content="5">\n' if auto_refresh else ""++ html_parts = ["""<!DOCTYPE html>+<html>+<head>+ <meta charset="utf-8">+""" + refresh_tag + """ <title>""" + title_prefix + """Skill Description Optimization</title>+ <link rel="preconnect" href="https://fonts.googleapis.com">+ <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>+ <link href="https://fonts.googleapis.com/css2?family=Poppins:wght@500;600&family=Lora:wght@400;500&display=swap" rel="stylesheet">+ <style>+ body {+ font-family: 'Lora', Georgia, serif;+ max-width: 100%;+ margin: 0 auto;+ padding: 20px;+ background: #faf9f5;+ color: #141413;+ }+ h1 { font-family: 'Poppins', sans-serif; color: #141413; }+ .explainer {+ background: white;+ padding: 15px;+ border-radius: 6px;+ margin-bottom: 20px;+ border: 1px solid #e8e6dc;+ color: #b0aea5;+ font-size: 0.875rem;+ line-height: 1.6;+ }+ .summary {+ background: white;+ padding: 15px;+ border-radius: 6px;+ margin-bottom: 20px;+ border: 1px solid #e8e6dc;+ }+ .summary p { margin: 5px 0; }+ .best { color: #788c5d; font-weight: bold; }+ .table-container {+ overflow-x: auto;+ width: 100%;+ }+ table {+ border-collapse: collapse;+ background: white;+ border: 1px solid #e8e6dc;+ border-radius: 6px;+ font-size: 12px;+ min-width: 100%;+ }+ th, td {+ padding: 8px;+ text-align: left;+ border: 1px solid #e8e6dc;+ white-space: normal;+ word-wrap: break-word;+ }+ th {+ font-family: 'Poppins', sans-serif;+ background: #141413;+ color: #faf9f5;+ font-weight: 500;+ }+ th.test-col {+ background: #6a9bcc;+ }+ th.query-col { min-width: 200px; }+ td.description {+ font-family: monospace;+ font-size: 11px;+ word-wrap: break-word;+ max-width: 400px;+ }+ td.result {+ text-align: center;+ font-size: 16px;+ min-width: 40px;+ }+ td.test-result {+ background: #f0f6fc;+ }+ .pass { color: #788c5d; }+ .fail { color: #c44; }+ .rate {+ font-size: 9px;+ color: #b0aea5;+ display: block;+ }+ tr:hover { background: #faf9f5; }+ .score {+ display: inline-block;+ padding: 2px 6px;+ border-radius: 4px;+ font-weight: bold;+ font-size: 11px;+ }+ .score-good { background: #eef2e8; color: #788c5d; }+ .score-ok { background: #fef3c7; color: #d97706; }+ .score-bad { background: #fceaea; color: #c44; }+ .train-label { color: #b0aea5; font-size: 10px; }+ .test-label { color: #6a9bcc; font-size: 10px; font-weight: bold; }+ .best-row { background: #f5f8f2; }+ th.positive-col { border-bottom: 3px solid #788c5d; }+ th.negative-col { border-bottom: 3px solid #c44; }+ th.test-col.positive-col { border-bottom: 3px solid #788c5d; }+ th.test-col.negative-col { border-bottom: 3px solid #c44; }+ .legend { font-family: 'Poppins', sans-serif; display: flex; gap: 20px; margin-bottom: 10px; font-size: 13px; align-items: center; }+ .legend-item { display: flex; align-items: center; gap: 6px; }+ .legend-swatch { width: 16px; height: 16px; border-radius: 3px; display: inline-block; }+ .swatch-positive { background: #141413; border-bottom: 3px solid #788c5d; }+ .swatch-negative { background: #141413; border-bottom: 3px solid #c44; }+ .swatch-test { background: #6a9bcc; }+ .swatch-train { background: #141413; }+ </style>+</head>+<body>+ <h1>""" + title_prefix + """Skill Description Optimization</h1>+ <div class="explainer">+ <strong>Optimizing your skill's description.</strong> This page updates automatically as Claude tests different versions of your skill's description. Each row is an iteration — a new description attempt. The columns show test queries: green checkmarks mean the skill triggered correctly (or correctly didn't trigger), red crosses mean it got it wrong. The "Train" score shows performance on queries used to improve the description; the "Test" score shows performance on held-out queries the optimizer hasn't seen. When it's done, Claude will apply the best-performing description to your skill.+ </div>+"""]++ # Summary section+ best_test_score = data.get('best_test_score')+ best_train_score = data.get('best_train_score')+ html_parts.append(f"""+ <div class="summary">+ <p><strong>Original:</strong> {html.escape(data.get('original_description', 'N/A'))}</p>+ <p class="best"><strong>Best:</strong> {html.escape(data.get('best_description', 'N/A'))}</p>+ <p><strong>Best Score:</strong> {data.get('best_score', 'N/A')} {'(test)' if best_test_score else '(train)'}</p>+ <p><strong>Iterations:</strong> {data.get('iterations_run', 0)} | <strong>Train:</strong> {data.get('train_size', '?')} | <strong>Test:</strong> {data.get('test_size', '?')}</p>+ </div>+""")++ # Legend+ html_parts.append("""+ <div class="legend">+ <span style="font-weight:600">Query columns:</span>+ <span class="legend-item"><span class="legend-swatch swatch-positive"></span> Should trigger</span>+ <span class="legend-item"><span class="legend-swatch swatch-negative"></span> Should NOT trigger</span>+ <span class="legend-item"><span class="legend-swatch swatch-train"></span> Train</span>+ <span class="legend-item"><span class="legend-swatch swatch-test"></span> Test</span>+ </div>+""")++ # Table header+ html_parts.append("""+ <div class="table-container">+ <table>+ <thead>+ <tr>+ <th>Iter</th>+ <th>Train</th>+ <th>Test</th>+ <th class="query-col">Description</th>+""")++ # Add column headers for train queries+ for qinfo in train_queries:+ polarity = "positive-col" if qinfo["should_trigger"] else "negative-col"+ html_parts.append(f' <th class="{polarity}">{html.escape(qinfo["query"])}</th>\n')++ # Add column headers for test queries (different color)+ for qinfo in test_queries:+ polarity = "positive-col" if qinfo["should_trigger"] else "negative-col"+ html_parts.append(f' <th class="test-col {polarity}">{html.escape(qinfo["query"])}</th>\n')++ html_parts.append(""" </tr>+ </thead>+ <tbody>+""")++ # Find best iteration for highlighting+ if test_queries:+ best_iter = max(history, key=lambda h: h.get("test_passed") or 0).get("iteration")+ else:+ best_iter = max(history, key=lambda h: h.get("train_passed", h.get("passed", 0))).get("iteration")++ # Add rows for each iteration+ for h in history:+ iteration = h.get("iteration", "?")+ train_passed = h.get("train_passed", h.get("passed", 0))+ train_total = h.get("train_total", h.get("total", 0))+ test_passed = h.get("test_passed")+ test_total = h.get("test_total")+ description = h.get("description", "")+ train_results = h.get("train_results", h.get("results", []))+ test_results = h.get("test_results", [])++ # Create lookups for results by query+ train_by_query = {r["query"]: r for r in train_results}+ test_by_query = {r["query"]: r for r in test_results} if test_results else {}++ # Compute aggregate correct/total runs across all retries+ def aggregate_runs(results: list[dict]) -> tuple[int, int]:+ correct = 0+ total = 0+ for r in results:+ runs = r.get("runs", 0)+ triggers = r.get("triggers", 0)+ total += runs+ if r.get("should_trigger", True):+ correct += triggers+ else:+ correct += runs - triggers+ return correct, total++ train_correct, train_runs = aggregate_runs(train_results)+ test_correct, test_runs = aggregate_runs(test_results)++ # Determine score classes+ def score_class(correct: int, total: int) -> str:+ if total > 0:+ ratio = correct / total+ if ratio >= 0.8:+ return "score-good"+ elif ratio >= 0.5:+ return "score-ok"+ return "score-bad"++ train_class = score_class(train_correct, train_runs)+ test_class = score_class(test_correct, test_runs)++ row_class = "best-row" if iteration == best_iter else ""++ html_parts.append(f""" <tr class="{row_class}">+ <td>{iteration}</td>+ <td><span class="score {train_class}">{train_correct}/{train_runs}</span></td>+ <td><span class="score {test_class}">{test_correct}/{test_runs}</span></td>+ <td class="description">{html.escape(description)}</td>+""")++ # Add result for each train query+ for qinfo in train_queries:+ r = train_by_query.get(qinfo["query"], {})+ did_pass = r.get("pass", False)+ triggers = r.get("triggers", 0)+ runs = r.get("runs", 0)++ icon = "✓" if did_pass else "✗"+ css_class = "pass" if did_pass else "fail"++ html_parts.append(f' <td class="result {css_class}">{icon}<span class="rate">{triggers}/{runs}</span></td>\n')++ # Add result for each test query (with different background)+ for qinfo in test_queries:+ r = test_by_query.get(qinfo["query"], {})+ did_pass = r.get("pass", False)+ triggers = r.get("triggers", 0)+ runs = r.get("runs", 0)++ icon = "✓" if did_pass else "✗"+ css_class = "pass" if did_pass else "fail"++ html_parts.append(f' <td class="result test-result {css_class}">{icon}<span class="rate">{triggers}/{runs}</span></td>\n')++ html_parts.append(" </tr>\n")++ html_parts.append(""" </tbody>+ </table>+ </div>+""")++ html_parts.append("""+</body>+</html>+""")++ return "".join(html_parts)+++def main():+ parser = argparse.ArgumentParser(description="Generate HTML report from run_loop output")+ parser.add_argument("input", help="Path to JSON output from run_loop.py (or - for stdin)")+ parser.add_argument("-o", "--output", default=None, help="Output HTML file (default: stdout)")+ parser.add_argument("--skill-name", default="", help="Skill name to include in the report title")+ args = parser.parse_args()++ if args.input == "-":+ data = json.load(sys.stdin)+ else:+ data = json.loads(Path(args.input).read_text())++ html_output = generate_html(data, skill_name=args.skill_name)++ if args.output:+ Path(args.output).write_text(html_output)+ print(f"Report written to {args.output}", file=sys.stderr)+ else:+ print(html_output)+++if __name__ == "__main__":+ main()diff --git a/.opencode/skills/skill-creator/scripts/improve_description.py b/.opencode/skills/skill-creator/scripts/improve_description.pynew file mode 100644index 0000000..06bcec7--- /dev/null+++ b/.opencode/skills/skill-creator/scripts/improve_description.py@@ -0,0 +1,247 @@+#!/usr/bin/env python3+"""Improve a skill description based on eval results.++Takes eval results (from run_eval.py) and generates an improved description+by calling `claude -p` as a subprocess (same auth pattern as run_eval.py —+uses the session's Claude Code auth, no separate ANTHROPIC_API_KEY needed).+"""++import argparse+import json+import os+import re+import subprocess+import sys+from pathlib import Path++from scripts.utils import parse_skill_md+++def _call_claude(prompt: str, model: str | None, timeout: int = 300) -> str:+ """Run `claude -p` with the prompt on stdin and return the text response.++ Prompt goes over stdin (not argv) because it embeds the full SKILL.md+ body and can easily exceed comfortable argv length.+ """+ cmd = ["claude", "-p", "--output-format", "text"]+ if model:+ cmd.extend(["--model", model])++ # Remove CLAUDECODE env var to allow nesting claude -p inside a+ # Claude Code session. The guard is for interactive terminal conflicts;+ # programmatic subprocess usage is safe. Same pattern as run_eval.py.+ env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}++ result = subprocess.run(+ cmd,+ input=prompt,+ capture_output=True,+ text=True,+ env=env,+ timeout=timeout,+ )+ if result.returncode != 0:+ raise RuntimeError(+ f"claude -p exited {result.returncode}\nstderr: {result.stderr}"+ )+ return result.stdout+++def improve_description(+ skill_name: str,+ skill_content: str,+ current_description: str,+ eval_results: dict,+ history: list[dict],+ model: str,+ test_results: dict | None = None,+ log_dir: Path | None = None,+ iteration: int | None = None,+) -> str:+ """Call Claude to improve the description based on eval results."""+ failed_triggers = [+ r for r in eval_results["results"]+ if r["should_trigger"] and not r["pass"]+ ]+ false_triggers = [+ r for r in eval_results["results"]+ if not r["should_trigger"] and not r["pass"]+ ]++ # Build scores summary+ train_score = f"{eval_results['summary']['passed']}/{eval_results['summary']['total']}"+ if test_results:+ test_score = f"{test_results['summary']['passed']}/{test_results['summary']['total']}"+ scores_summary = f"Train: {train_score}, Test: {test_score}"+ else:+ scores_summary = f"Train: {train_score}"++ prompt = f"""You are optimizing a skill description for a Claude Code skill called "{skill_name}". A "skill" is sort of like a prompt, but with progressive disclosure -- there's a title and description that Claude sees when deciding whether to use the skill, and then if it does use the skill, it reads the .md file which has lots more details and potentially links to other resources in the skill folder like helper files and scripts and additional documentation or examples.++The description appears in Claude's "available_skills" list. When a user sends a query, Claude decides whether to invoke the skill based solely on the title and on this description. Your goal is to write a description that triggers for relevant queries, and doesn't trigger for irrelevant ones.++Here's the current description:+<current_description>+"{current_description}"+</current_description>++Current scores ({scores_summary}):+<scores_summary>+"""+ if failed_triggers:+ prompt += "FAILED TO TRIGGER (should have triggered but didn't):\n"+ for r in failed_triggers:+ prompt += f' - "{r["query"]}" (triggered {r["triggers"]}/{r["runs"]} times)\n'+ prompt += "\n"++ if false_triggers:+ prompt += "FALSE TRIGGERS (triggered but shouldn't have):\n"+ for r in false_triggers:+ prompt += f' - "{r["query"]}" (triggered {r["triggers"]}/{r["runs"]} times)\n'+ prompt += "\n"++ if history:+ prompt += "PREVIOUS ATTEMPTS (do NOT repeat these — try something structurally different):\n\n"+ for h in history:+ train_s = f"{h.get('train_passed', h.get('passed', 0))}/{h.get('train_total', h.get('total', 0))}"+ test_s = f"{h.get('test_passed', '?')}/{h.get('test_total', '?')}" if h.get('test_passed') is not None else None+ score_str = f"train={train_s}" + (f", test={test_s}" if test_s else "")+ prompt += f'<attempt {score_str}>\n'+ prompt += f'Description: "{h["description"]}"\n'+ if "results" in h:+ prompt += "Train results:\n"+ for r in h["results"]:+ status = "PASS" if r["pass"] else "FAIL"+ prompt += f' [{status}] "{r["query"][:80]}" (triggered {r["triggers"]}/{r["runs"]})\n'+ if h.get("note"):+ prompt += f'Note: {h["note"]}\n'+ prompt += "</attempt>\n\n"++ prompt += f"""</scores_summary>++Skill content (for context on what the skill does):+<skill_content>+{skill_content}+</skill_content>++Based on the failures, write a new and improved description that is more likely to trigger correctly. When I say "based on the failures", it's a bit of a tricky line to walk because we don't want to overfit to the specific cases you're seeing. So what I DON'T want you to do is produce an ever-expanding list of specific queries that this skill should or shouldn't trigger for. Instead, try to generalize from the failures to broader categories of user intent and situations where this skill would be useful or not useful. The reason for this is twofold:++1. Avoid overfitting+2. The list might get loooong and it's injected into ALL queries and there might be a lot of skills, so we don't want to blow too much space on any given description.++Concretely, your description should not be more than about 100-200 words, even if that comes at the cost of accuracy. There is a hard limit of 1024 characters — descriptions over that will be truncated, so stay comfortably under it.++Here are some tips that we've found to work well in writing these descriptions:+- The skill should be phrased in the imperative -- "Use this skill for" rather than "this skill does"+- The skill description should focus on the user's intent, what they are trying to achieve, vs. the implementation details of how the skill works.+- The description competes with other skills for Claude's attention — make it distinctive and immediately recognizable.+- If you're getting lots of failures after repeated attempts, change things up. Try different sentence structures or wordings.++I'd encourage you to be creative and mix up the style in different iterations since you'll have multiple opportunities to try different approaches and we'll just grab the highest-scoring one at the end.++Please respond with only the new description text in <new_description> tags, nothing else."""++ text = _call_claude(prompt, model)++ match = re.search(r"<new_description>(.*?)</new_description>", text, re.DOTALL)+ description = match.group(1).strip().strip('"') if match else text.strip().strip('"')++ transcript: dict = {+ "iteration": iteration,+ "prompt": prompt,+ "response": text,+ "parsed_description": description,+ "char_count": len(description),+ "over_limit": len(description) > 1024,+ }++ # Safety net: the prompt already states the 1024-char hard limit, but if+ # the model blew past it anyway, make one fresh single-turn call that+ # quotes the too-long version and asks for a shorter rewrite. (The old+ # SDK path did this as a true multi-turn; `claude -p` is one-shot, so we+ # inline the prior output into the new prompt instead.)+ if len(description) > 1024:+ shorten_prompt = (+ f"{prompt}\n\n"+ f"---\n\n"+ f"A previous attempt produced this description, which at "+ f"{len(description)} characters is over the 1024-character hard limit:\n\n"+ f'"{description}"\n\n'+ f"Rewrite it to be under 1024 characters while keeping the most "+ f"important trigger words and intent coverage. Respond with only "+ f"the new description in <new_description> tags."+ )+ shorten_text = _call_claude(shorten_prompt, model)+ match = re.search(r"<new_description>(.*?)</new_description>", shorten_text, re.DOTALL)+ shortened = match.group(1).strip().strip('"') if match else shorten_text.strip().strip('"')++ transcript["rewrite_prompt"] = shorten_prompt+ transcript["rewrite_response"] = shorten_text+ transcript["rewrite_description"] = shortened+ transcript["rewrite_char_count"] = len(shortened)+ description = shortened++ transcript["final_description"] = description++ if log_dir:+ log_dir.mkdir(parents=True, exist_ok=True)+ log_file = log_dir / f"improve_iter_{iteration or 'unknown'}.json"+ log_file.write_text(json.dumps(transcript, indent=2))++ return description+++def main():+ parser = argparse.ArgumentParser(description="Improve a skill description based on eval results")+ parser.add_argument("--eval-results", required=True, help="Path to eval results JSON (from run_eval.py)")+ parser.add_argument("--skill-path", required=True, help="Path to skill directory")+ parser.add_argument("--history", default=None, help="Path to history JSON (previous attempts)")+ parser.add_argument("--model", required=True, help="Model for improvement")+ parser.add_argument("--verbose", action="store_true", help="Print thinking to stderr")+ args = parser.parse_args()++ skill_path = Path(args.skill_path)+ if not (skill_path / "SKILL.md").exists():+ print(f"Error: No SKILL.md found at {skill_path}", file=sys.stderr)+ sys.exit(1)++ eval_results = json.loads(Path(args.eval_results).read_text())+ history = []+ if args.history:+ history = json.loads(Path(args.history).read_text())++ name, _, content = parse_skill_md(skill_path)+ current_description = eval_results["description"]++ if args.verbose:+ print(f"Current: {current_description}", file=sys.stderr)+ print(f"Score: {eval_results['summary']['passed']}/{eval_results['summary']['total']}", file=sys.stderr)++ new_description = improve_description(+ skill_name=name,+ skill_content=content,+ current_description=current_description,+ eval_results=eval_results,+ history=history,+ model=args.model,+ )++ if args.verbose:+ print(f"Improved: {new_description}", file=sys.stderr)++ # Output as JSON with both the new description and updated history+ output = {+ "description": new_description,+ "history": history + [{+ "description": current_description,+ "passed": eval_results["summary"]["passed"],+ "failed": eval_results["summary"]["failed"],+ "total": eval_results["summary"]["total"],+ "results": eval_results["results"],+ }],+ }+ print(json.dumps(output, indent=2))+++if __name__ == "__main__":+ main()diff --git a/.opencode/skills/skill-creator/scripts/package_skill.py b/.opencode/skills/skill-creator/scripts/package_skill.pynew file mode 100644index 0000000..f48eac4--- /dev/null+++ b/.opencode/skills/skill-creator/scripts/package_skill.py@@ -0,0 +1,136 @@+#!/usr/bin/env python3+"""+Skill Packager - Creates a distributable .skill file of a skill folder++Usage:+ python utils/package_skill.py <path/to/skill-folder> [output-directory]++Example:+ python utils/package_skill.py skills/public/my-skill+ python utils/package_skill.py skills/public/my-skill ./dist+"""++import fnmatch+import sys+import zipfile+from pathlib import Path+from scripts.quick_validate import validate_skill++# Patterns to exclude when packaging skills.+EXCLUDE_DIRS = {"__pycache__", "node_modules"}+EXCLUDE_GLOBS = {"*.pyc"}+EXCLUDE_FILES = {".DS_Store"}+# Directories excluded only at the skill root (not when nested deeper).+ROOT_EXCLUDE_DIRS = {"evals"}+++def should_exclude(rel_path: Path) -> bool:+ """Check if a path should be excluded from packaging."""+ parts = rel_path.parts+ if any(part in EXCLUDE_DIRS for part in parts):+ return True+ # rel_path is relative to skill_path.parent, so parts[0] is the skill+ # folder name and parts[1] (if present) is the first subdir.+ if len(parts) > 1 and parts[1] in ROOT_EXCLUDE_DIRS:+ return True+ name = rel_path.name+ if name in EXCLUDE_FILES:+ return True+ return any(fnmatch.fnmatch(name, pat) for pat in EXCLUDE_GLOBS)+++def package_skill(skill_path, output_dir=None):+ """+ Package a skill folder into a .skill file.++ Args:+ skill_path: Path to the skill folder+ output_dir: Optional output directory for the .skill file (defaults to current directory)++ Returns:+ Path to the created .skill file, or None if error+ """+ skill_path = Path(skill_path).resolve()++ # Validate skill folder exists+ if not skill_path.exists():+ print(f"❌ Error: Skill folder not found: {skill_path}")+ return None++ if not skill_path.is_dir():+ print(f"❌ Error: Path is not a directory: {skill_path}")+ return None++ # Validate SKILL.md exists+ skill_md = skill_path / "SKILL.md"+ if not skill_md.exists():+ print(f"❌ Error: SKILL.md not found in {skill_path}")+ return None++ # Run validation before packaging+ print("🔍 Validating skill...")+ valid, message = validate_skill(skill_path)+ if not valid:+ print(f"❌ Validation failed: {message}")+ print(" Please fix the validation errors before packaging.")+ return None+ print(f"✅ {message}\n")++ # Determine output location+ skill_name = skill_path.name+ if output_dir:+ output_path = Path(output_dir).resolve()+ output_path.mkdir(parents=True, exist_ok=True)+ else:+ output_path = Path.cwd()++ skill_filename = output_path / f"{skill_name}.skill"++ # Create the .skill file (zip format)+ try:+ with zipfile.ZipFile(skill_filename, 'w', zipfile.ZIP_DEFLATED) as zipf:+ # Walk through the skill directory, excluding build artifacts+ for file_path in skill_path.rglob('*'):+ if not file_path.is_file():+ continue+ arcname = file_path.relative_to(skill_path.parent)+ if should_exclude(arcname):+ print(f" Skipped: {arcname}")+ continue+ zipf.write(file_path, arcname)+ print(f" Added: {arcname}")++ print(f"\n✅ Successfully packaged skill to: {skill_filename}")+ return skill_filename++ except Exception as e:+ print(f"❌ Error creating .skill file: {e}")+ return None+++def main():+ if len(sys.argv) < 2:+ print("Usage: python utils/package_skill.py <path/to/skill-folder> [output-directory]")+ print("\nExample:")+ print(" python utils/package_skill.py skills/public/my-skill")+ print(" python utils/package_skill.py skills/public/my-skill ./dist")+ sys.exit(1)++ skill_path = sys.argv[1]+ output_dir = sys.argv[2] if len(sys.argv) > 2 else None++ print(f"📦 Packaging skill: {skill_path}")+ if output_dir:+ print(f" Output directory: {output_dir}")+ print()++ result = package_skill(skill_path, output_dir)++ if result:+ sys.exit(0)+ else:+ sys.exit(1)+++if __name__ == "__main__":+ main()diff --git a/.opencode/skills/skill-creator/scripts/quick_validate.py b/.opencode/skills/skill-creator/scripts/quick_validate.pynew file mode 100644index 0000000..ed8e1dd--- /dev/null+++ b/.opencode/skills/skill-creator/scripts/quick_validate.py@@ -0,0 +1,103 @@+#!/usr/bin/env python3+"""+Quick validation script for skills - minimal version+"""++import sys+import os+import re+import yaml+from pathlib import Path++def validate_skill(skill_path):+ """Basic validation of a skill"""+ skill_path = Path(skill_path)++ # Check SKILL.md exists+ skill_md = skill_path / 'SKILL.md'+ if not skill_md.exists():+ return False, "SKILL.md not found"++ # Read and validate frontmatter+ content = skill_md.read_text()+ if not content.startswith('---'):+ return False, "No YAML frontmatter found"++ # Extract frontmatter+ match = re.match(r'^---\n(.*?)\n---', content, re.DOTALL)+ if not match:+ return False, "Invalid frontmatter format"++ frontmatter_text = match.group(1)++ # Parse YAML frontmatter+ try:+ frontmatter = yaml.safe_load(frontmatter_text)+ if not isinstance(frontmatter, dict):+ return False, "Frontmatter must be a YAML dictionary"+ except yaml.YAMLError as e:+ return False, f"Invalid YAML in frontmatter: {e}"++ # Define allowed properties+ ALLOWED_PROPERTIES = {'name', 'description', 'license', 'allowed-tools', 'metadata', 'compatibility'}++ # Check for unexpected properties (excluding nested keys under metadata)+ unexpected_keys = set(frontmatter.keys()) - ALLOWED_PROPERTIES+ if unexpected_keys:+ return False, (+ f"Unexpected key(s) in SKILL.md frontmatter: {', '.join(sorted(unexpected_keys))}. "+ f"Allowed properties are: {', '.join(sorted(ALLOWED_PROPERTIES))}"+ )++ # Check required fields+ if 'name' not in frontmatter:+ return False, "Missing 'name' in frontmatter"+ if 'description' not in frontmatter:+ return False, "Missing 'description' in frontmatter"++ # Extract name for validation+ name = frontmatter.get('name', '')+ if not isinstance(name, str):+ return False, f"Name must be a string, got {type(name).__name__}"+ name = name.strip()+ if name:+ # Check naming convention (kebab-case: lowercase with hyphens)+ if not re.match(r'^[a-z0-9-]+$', name):+ return False, f"Name '{name}' should be kebab-case (lowercase letters, digits, and hyphens only)"+ if name.startswith('-') or name.endswith('-') or '--' in name:+ return False, f"Name '{name}' cannot start/end with hyphen or contain consecutive hyphens"+ # Check name length (max 64 characters per spec)+ if len(name) > 64:+ return False, f"Name is too long ({len(name)} characters). Maximum is 64 characters."++ # Extract and validate description+ description = frontmatter.get('description', '')+ if not isinstance(description, str):+ return False, f"Description must be a string, got {type(description).__name__}"+ description = description.strip()+ if description:+ # Check for angle brackets+ if '<' in description or '>' in description:+ return False, "Description cannot contain angle brackets (< or >)"+ # Check description length (max 1024 characters per spec)+ if len(description) > 1024:+ return False, f"Description is too long ({len(description)} characters). Maximum is 1024 characters."++ # Validate compatibility field if present (optional)+ compatibility = frontmatter.get('compatibility', '')+ if compatibility:+ if not isinstance(compatibility, str):+ return False, f"Compatibility must be a string, got {type(compatibility).__name__}"+ if len(compatibility) > 500:+ return False, f"Compatibility is too long ({len(compatibility)} characters). Maximum is 500 characters."++ return True, "Skill is valid!"++if __name__ == "__main__":+ if len(sys.argv) != 2:+ print("Usage: python quick_validate.py <skill_directory>")+ sys.exit(1)++ valid, message = validate_skill(sys.argv[1])+ print(message)+ sys.exit(0 if valid else 1)\ No newline at end of filediff --git a/.opencode/skills/skill-creator/scripts/run_eval.py b/.opencode/skills/skill-creator/scripts/run_eval.pynew file mode 100644index 0000000..e58c70b--- /dev/null+++ b/.opencode/skills/skill-creator/scripts/run_eval.py@@ -0,0 +1,310 @@+#!/usr/bin/env python3+"""Run trigger evaluation for a skill description.++Tests whether a skill's description causes Claude to trigger (read the skill)+for a set of queries. Outputs results as JSON.+"""++import argparse+import json+import os+import select+import subprocess+import sys+import time+import uuid+from concurrent.futures import ProcessPoolExecutor, as_completed+from pathlib import Path++from scripts.utils import parse_skill_md+++def find_project_root() -> Path:+ """Find the project root by walking up from cwd looking for .claude/.++ Mimics how Claude Code discovers its project root, so the command file+ we create ends up where claude -p will look for it.+ """+ current = Path.cwd()+ for parent in [current, *current.parents]:+ if (parent / ".claude").is_dir():+ return parent+ return current+++def run_single_query(+ query: str,+ skill_name: str,+ skill_description: str,+ timeout: int,+ project_root: str,+ model: str | None = None,+) -> bool:+ """Run a single query and return whether the skill was triggered.++ Creates a command file in .claude/commands/ so it appears in Claude's+ available_skills list, then runs `claude -p` with the raw query.+ Uses --include-partial-messages to detect triggering early from+ stream events (content_block_start) rather than waiting for the+ full assistant message, which only arrives after tool execution.+ """+ unique_id = uuid.uuid4().hex[:8]+ clean_name = f"{skill_name}-skill-{unique_id}"+ project_commands_dir = Path(project_root) / ".claude" / "commands"+ command_file = project_commands_dir / f"{clean_name}.md"++ try:+ project_commands_dir.mkdir(parents=True, exist_ok=True)+ # Use YAML block scalar to avoid breaking on quotes in description+ indented_desc = "\n ".join(skill_description.split("\n"))+ command_content = (+ f"---\n"+ f"description: |\n"+ f" {indented_desc}\n"+ f"---\n\n"+ f"# {skill_name}\n\n"+ f"This skill handles: {skill_description}\n"+ )+ command_file.write_text(command_content)++ cmd = [+ "claude",+ "-p", query,+ "--output-format", "stream-json",+ "--verbose",+ "--include-partial-messages",+ ]+ if model:+ cmd.extend(["--model", model])++ # Remove CLAUDECODE env var to allow nesting claude -p inside a+ # Claude Code session. The guard is for interactive terminal conflicts;+ # programmatic subprocess usage is safe.+ env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}++ process = subprocess.Popen(+ cmd,+ stdout=subprocess.PIPE,+ stderr=subprocess.DEVNULL,+ cwd=project_root,+ env=env,+ )++ triggered = False+ start_time = time.time()+ buffer = ""+ # Track state for stream event detection+ pending_tool_name = None+ accumulated_json = ""++ try:+ while time.time() - start_time < timeout:+ if process.poll() is not None:+ remaining = process.stdout.read()+ if remaining:+ buffer += remaining.decode("utf-8", errors="replace")+ break++ ready, _, _ = select.select([process.stdout], [], [], 1.0)+ if not ready:+ continue++ chunk = os.read(process.stdout.fileno(), 8192)+ if not chunk:+ break+ buffer += chunk.decode("utf-8", errors="replace")++ while "\n" in buffer:+ line, buffer = buffer.split("\n", 1)+ line = line.strip()+ if not line:+ continue++ try:+ event = json.loads(line)+ except json.JSONDecodeError:+ continue++ # Early detection via stream events+ if event.get("type") == "stream_event":+ se = event.get("event", {})+ se_type = se.get("type", "")++ if se_type == "content_block_start":+ cb = se.get("content_block", {})+ if cb.get("type") == "tool_use":+ tool_name = cb.get("name", "")+ if tool_name in ("Skill", "Read"):+ pending_tool_name = tool_name+ accumulated_json = ""+ else:+ return False++ elif se_type == "content_block_delta" and pending_tool_name:+ delta = se.get("delta", {})+ if delta.get("type") == "input_json_delta":+ accumulated_json += delta.get("partial_json", "")+ if clean_name in accumulated_json:+ return True++ elif se_type in ("content_block_stop", "message_stop"):+ if pending_tool_name:+ return clean_name in accumulated_json+ if se_type == "message_stop":+ return False++ # Fallback: full assistant message+ elif event.get("type") == "assistant":+ message = event.get("message", {})+ for content_item in message.get("content", []):+ if content_item.get("type") != "tool_use":+ continue+ tool_name = content_item.get("name", "")+ tool_input = content_item.get("input", {})+ if tool_name == "Skill" and clean_name in tool_input.get("skill", ""):+ triggered = True+ elif tool_name == "Read" and clean_name in tool_input.get("file_path", ""):+ triggered = True+ return triggered++ elif event.get("type") == "result":+ return triggered+ finally:+ # Clean up process on any exit path (return, exception, timeout)+ if process.poll() is None:+ process.kill()+ process.wait()++ return triggered+ finally:+ if command_file.exists():+ command_file.unlink()+++def run_eval(+ eval_set: list[dict],+ skill_name: str,+ description: str,+ num_workers: int,+ timeout: int,+ project_root: Path,+ runs_per_query: int = 1,+ trigger_threshold: float = 0.5,+ model: str | None = None,+) -> dict:+ """Run the full eval set and return results."""+ results = []++ with ProcessPoolExecutor(max_workers=num_workers) as executor:+ future_to_info = {}+ for item in eval_set:+ for run_idx in range(runs_per_query):+ future = executor.submit(+ run_single_query,+ item["query"],+ skill_name,+ description,+ timeout,+ str(project_root),+ model,+ )+ future_to_info[future] = (item, run_idx)++ query_triggers: dict[str, list[bool]] = {}+ query_items: dict[str, dict] = {}+ for future in as_completed(future_to_info):+ item, _ = future_to_info[future]+ query = item["query"]+ query_items[query] = item+ if query not in query_triggers:+ query_triggers[query] = []+ try:+ query_triggers[query].append(future.result())+ except Exception as e:+ print(f"Warning: query failed: {e}", file=sys.stderr)+ query_triggers[query].append(False)++ for query, triggers in query_triggers.items():+ item = query_items[query]+ trigger_rate = sum(triggers) / len(triggers)+ should_trigger = item["should_trigger"]+ if should_trigger:+ did_pass = trigger_rate >= trigger_threshold+ else:+ did_pass = trigger_rate < trigger_threshold+ results.append({+ "query": query,+ "should_trigger": should_trigger,+ "trigger_rate": trigger_rate,+ "triggers": sum(triggers),+ "runs": len(triggers),+ "pass": did_pass,+ })++ passed = sum(1 for r in results if r["pass"])+ total = len(results)++ return {+ "skill_name": skill_name,+ "description": description,+ "results": results,+ "summary": {+ "total": total,+ "passed": passed,+ "failed": total - passed,+ },+ }+++def main():+ parser = argparse.ArgumentParser(description="Run trigger evaluation for a skill description")+ parser.add_argument("--eval-set", required=True, help="Path to eval set JSON file")+ parser.add_argument("--skill-path", required=True, help="Path to skill directory")+ parser.add_argument("--description", default=None, help="Override description to test")+ parser.add_argument("--num-workers", type=int, default=10, help="Number of parallel workers")+ parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds")+ parser.add_argument("--runs-per-query", type=int, default=3, help="Number of runs per query")+ parser.add_argument("--trigger-threshold", type=float, default=0.5, help="Trigger rate threshold")+ parser.add_argument("--model", default=None, help="Model to use for claude -p (default: user's configured model)")+ parser.add_argument("--verbose", action="store_true", help="Print progress to stderr")+ args = parser.parse_args()++ eval_set = json.loads(Path(args.eval_set).read_text())+ skill_path = Path(args.skill_path)++ if not (skill_path / "SKILL.md").exists():+ print(f"Error: No SKILL.md found at {skill_path}", file=sys.stderr)+ sys.exit(1)++ name, original_description, content = parse_skill_md(skill_path)+ description = args.description or original_description+ project_root = find_project_root()++ if args.verbose:+ print(f"Evaluating: {description}", file=sys.stderr)++ output = run_eval(+ eval_set=eval_set,+ skill_name=name,+ description=description,+ num_workers=args.num_workers,+ timeout=args.timeout,+ project_root=project_root,+ runs_per_query=args.runs_per_query,+ trigger_threshold=args.trigger_threshold,+ model=args.model,+ )++ if args.verbose:+ summary = output["summary"]+ print(f"Results: {summary['passed']}/{summary['total']} passed", file=sys.stderr)+ for r in output["results"]:+ status = "PASS" if r["pass"] else "FAIL"+ rate_str = f"{r['triggers']}/{r['runs']}"+ print(f" [{status}] rate={rate_str} expected={r['should_trigger']}: {r['query'][:70]}", file=sys.stderr)++ print(json.dumps(output, indent=2))+++if __name__ == "__main__":+ main()diff --git a/.opencode/skills/skill-creator/scripts/run_loop.py b/.opencode/skills/skill-creator/scripts/run_loop.pynew file mode 100644index 0000000..30a263d--- /dev/null+++ b/.opencode/skills/skill-creator/scripts/run_loop.py@@ -0,0 +1,328 @@+#!/usr/bin/env python3+"""Run the eval + improve loop until all pass or max iterations reached.++Combines run_eval.py and improve_description.py in a loop, tracking history+and returning the best description found. Supports train/test split to prevent+overfitting.+"""++import argparse+import json+import random+import sys+import tempfile+import time+import webbrowser+from pathlib import Path++from scripts.generate_report import generate_html+from scripts.improve_description import improve_description+from scripts.run_eval import find_project_root, run_eval+from scripts.utils import parse_skill_md+++def split_eval_set(eval_set: list[dict], holdout: float, seed: int = 42) -> tuple[list[dict], list[dict]]:+ """Split eval set into train and test sets, stratified by should_trigger."""+ random.seed(seed)++ # Separate by should_trigger+ trigger = [e for e in eval_set if e["should_trigger"]]+ no_trigger = [e for e in eval_set if not e["should_trigger"]]++ # Shuffle each group+ random.shuffle(trigger)+ random.shuffle(no_trigger)++ # Calculate split points+ n_trigger_test = max(1, int(len(trigger) * holdout))+ n_no_trigger_test = max(1, int(len(no_trigger) * holdout))++ # Split+ test_set = trigger[:n_trigger_test] + no_trigger[:n_no_trigger_test]+ train_set = trigger[n_trigger_test:] + no_trigger[n_no_trigger_test:]++ return train_set, test_set+++def run_loop(+ eval_set: list[dict],+ skill_path: Path,+ description_override: str | None,+ num_workers: int,+ timeout: int,+ max_iterations: int,+ runs_per_query: int,+ trigger_threshold: float,+ holdout: float,+ model: str,+ verbose: bool,+ live_report_path: Path | None = None,+ log_dir: Path | None = None,+) -> dict:+ """Run the eval + improvement loop."""+ project_root = find_project_root()+ name, original_description, content = parse_skill_md(skill_path)+ current_description = description_override or original_description++ # Split into train/test if holdout > 0+ if holdout > 0:+ train_set, test_set = split_eval_set(eval_set, holdout)+ if verbose:+ print(f"Split: {len(train_set)} train, {len(test_set)} test (holdout={holdout})", file=sys.stderr)+ else:+ train_set = eval_set+ test_set = []++ history = []+ exit_reason = "unknown"++ for iteration in range(1, max_iterations + 1):+ if verbose:+ print(f"\n{'='*60}", file=sys.stderr)+ print(f"Iteration {iteration}/{max_iterations}", file=sys.stderr)+ print(f"Description: {current_description}", file=sys.stderr)+ print(f"{'='*60}", file=sys.stderr)++ # Evaluate train + test together in one batch for parallelism+ all_queries = train_set + test_set+ t0 = time.time()+ all_results = run_eval(+ eval_set=all_queries,+ skill_name=name,+ description=current_description,+ num_workers=num_workers,+ timeout=timeout,+ project_root=project_root,+ runs_per_query=runs_per_query,+ trigger_threshold=trigger_threshold,+ model=model,+ )+ eval_elapsed = time.time() - t0++ # Split results back into train/test by matching queries+ train_queries_set = {q["query"] for q in train_set}+ train_result_list = [r for r in all_results["results"] if r["query"] in train_queries_set]+ test_result_list = [r for r in all_results["results"] if r["query"] not in train_queries_set]++ train_passed = sum(1 for r in train_result_list if r["pass"])+ train_total = len(train_result_list)+ train_summary = {"passed": train_passed, "failed": train_total - train_passed, "total": train_total}+ train_results = {"results": train_result_list, "summary": train_summary}++ if test_set:+ test_passed = sum(1 for r in test_result_list if r["pass"])+ test_total = len(test_result_list)+ test_summary = {"passed": test_passed, "failed": test_total - test_passed, "total": test_total}+ test_results = {"results": test_result_list, "summary": test_summary}+ else:+ test_results = None+ test_summary = None++ history.append({+ "iteration": iteration,+ "description": current_description,+ "train_passed": train_summary["passed"],+ "train_failed": train_summary["failed"],+ "train_total": train_summary["total"],+ "train_results": train_results["results"],+ "test_passed": test_summary["passed"] if test_summary else None,+ "test_failed": test_summary["failed"] if test_summary else None,+ "test_total": test_summary["total"] if test_summary else None,+ "test_results": test_results["results"] if test_results else None,+ # For backward compat with report generator+ "passed": train_summary["passed"],+ "failed": train_summary["failed"],+ "total": train_summary["total"],+ "results": train_results["results"],+ })++ # Write live report if path provided+ if live_report_path:+ partial_output = {+ "original_description": original_description,+ "best_description": current_description,+ "best_score": "in progress",+ "iterations_run": len(history),+ "holdout": holdout,+ "train_size": len(train_set),+ "test_size": len(test_set),+ "history": history,+ }+ live_report_path.write_text(generate_html(partial_output, auto_refresh=True, skill_name=name))++ if verbose:+ def print_eval_stats(label, results, elapsed):+ pos = [r for r in results if r["should_trigger"]]+ neg = [r for r in results if not r["should_trigger"]]+ tp = sum(r["triggers"] for r in pos)+ pos_runs = sum(r["runs"] for r in pos)+ fn = pos_runs - tp+ fp = sum(r["triggers"] for r in neg)+ neg_runs = sum(r["runs"] for r in neg)+ tn = neg_runs - fp+ total = tp + tn + fp + fn+ precision = tp / (tp + fp) if (tp + fp) > 0 else 1.0+ recall = tp / (tp + fn) if (tp + fn) > 0 else 1.0+ accuracy = (tp + tn) / total if total > 0 else 0.0+ print(f"{label}: {tp+tn}/{total} correct, precision={precision:.0%} recall={recall:.0%} accuracy={accuracy:.0%} ({elapsed:.1f}s)", file=sys.stderr)+ for r in results:+ status = "PASS" if r["pass"] else "FAIL"+ rate_str = f"{r['triggers']}/{r['runs']}"+ print(f" [{status}] rate={rate_str} expected={r['should_trigger']}: {r['query'][:60]}", file=sys.stderr)++ print_eval_stats("Train", train_results["results"], eval_elapsed)+ if test_summary:+ print_eval_stats("Test ", test_results["results"], 0)++ if train_summary["failed"] == 0:+ exit_reason = f"all_passed (iteration {iteration})"+ if verbose:+ print(f"\nAll train queries passed on iteration {iteration}!", file=sys.stderr)+ break++ if iteration == max_iterations:+ exit_reason = f"max_iterations ({max_iterations})"+ if verbose:+ print(f"\nMax iterations reached ({max_iterations}).", file=sys.stderr)+ break++ # Improve the description based on train results+ if verbose:+ print(f"\nImproving description...", file=sys.stderr)++ t0 = time.time()+ # Strip test scores from history so improvement model can't see them+ blinded_history = [+ {k: v for k, v in h.items() if not k.startswith("test_")}+ for h in history+ ]+ new_description = improve_description(+ skill_name=name,+ skill_content=content,+ current_description=current_description,+ eval_results=train_results,+ history=blinded_history,+ model=model,+ log_dir=log_dir,+ iteration=iteration,+ )+ improve_elapsed = time.time() - t0++ if verbose:+ print(f"Proposed ({improve_elapsed:.1f}s): {new_description}", file=sys.stderr)++ current_description = new_description++ # Find the best iteration by TEST score (or train if no test set)+ if test_set:+ best = max(history, key=lambda h: h["test_passed"] or 0)+ best_score = f"{best['test_passed']}/{best['test_total']}"+ else:+ best = max(history, key=lambda h: h["train_passed"])+ best_score = f"{best['train_passed']}/{best['train_total']}"++ if verbose:+ print(f"\nExit reason: {exit_reason}", file=sys.stderr)+ print(f"Best score: {best_score} (iteration {best['iteration']})", file=sys.stderr)++ return {+ "exit_reason": exit_reason,+ "original_description": original_description,+ "best_description": best["description"],+ "best_score": best_score,+ "best_train_score": f"{best['train_passed']}/{best['train_total']}",+ "best_test_score": f"{best['test_passed']}/{best['test_total']}" if test_set else None,+ "final_description": current_description,+ "iterations_run": len(history),+ "holdout": holdout,+ "train_size": len(train_set),+ "test_size": len(test_set),+ "history": history,+ }+++def main():+ parser = argparse.ArgumentParser(description="Run eval + improve loop")+ parser.add_argument("--eval-set", required=True, help="Path to eval set JSON file")+ parser.add_argument("--skill-path", required=True, help="Path to skill directory")+ parser.add_argument("--description", default=None, help="Override starting description")+ parser.add_argument("--num-workers", type=int, default=10, help="Number of parallel workers")+ parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds")+ parser.add_argument("--max-iterations", type=int, default=5, help="Max improvement iterations")+ parser.add_argument("--runs-per-query", type=int, default=3, help="Number of runs per query")+ parser.add_argument("--trigger-threshold", type=float, default=0.5, help="Trigger rate threshold")+ parser.add_argument("--holdout", type=float, default=0.4, help="Fraction of eval set to hold out for testing (0 to disable)")+ parser.add_argument("--model", required=True, help="Model for improvement")+ parser.add_argument("--verbose", action="store_true", help="Print progress to stderr")+ parser.add_argument("--report", default="auto", help="Generate HTML report at this path (default: 'auto' for temp file, 'none' to disable)")+ parser.add_argument("--results-dir", default=None, help="Save all outputs (results.json, report.html, log.txt) to a timestamped subdirectory here")+ args = parser.parse_args()++ eval_set = json.loads(Path(args.eval_set).read_text())+ skill_path = Path(args.skill_path)++ if not (skill_path / "SKILL.md").exists():+ print(f"Error: No SKILL.md found at {skill_path}", file=sys.stderr)+ sys.exit(1)++ name, _, _ = parse_skill_md(skill_path)++ # Set up live report path+ if args.report != "none":+ if args.report == "auto":+ timestamp = time.strftime("%Y%m%d_%H%M%S")+ live_report_path = Path(tempfile.gettempdir()) / f"skill_description_report_{skill_path.name}_{timestamp}.html"+ else:+ live_report_path = Path(args.report)+ # Open the report immediately so the user can watch+ live_report_path.write_text("<html><body><h1>Starting optimization loop...</h1><meta http-equiv='refresh' content='5'></body></html>")+ webbrowser.open(str(live_report_path))+ else:+ live_report_path = None++ # Determine output directory (create before run_loop so logs can be written)+ if args.results_dir:+ timestamp = time.strftime("%Y-%m-%d_%H%M%S")+ results_dir = Path(args.results_dir) / timestamp+ results_dir.mkdir(parents=True, exist_ok=True)+ else:+ results_dir = None++ log_dir = results_dir / "logs" if results_dir else None++ output = run_loop(+ eval_set=eval_set,+ skill_path=skill_path,+ description_override=args.description,+ num_workers=args.num_workers,+ timeout=args.timeout,+ max_iterations=args.max_iterations,+ runs_per_query=args.runs_per_query,+ trigger_threshold=args.trigger_threshold,+ holdout=args.holdout,+ model=args.model,+ verbose=args.verbose,+ live_report_path=live_report_path,+ log_dir=log_dir,+ )++ # Save JSON output+ json_output = json.dumps(output, indent=2)+ print(json_output)+ if results_dir:+ (results_dir / "results.json").write_text(json_output)++ # Write final HTML report (without auto-refresh)+ if live_report_path:+ live_report_path.write_text(generate_html(output, auto_refresh=False, skill_name=name))+ print(f"\nReport: {live_report_path}", file=sys.stderr)++ if results_dir and live_report_path:+ (results_dir / "report.html").write_text(generate_html(output, auto_refresh=False, skill_name=name))++ if results_dir:+ print(f"Results saved to: {results_dir}", file=sys.stderr)+++if __name__ == "__main__":+ main()diff --git a/.opencode/skills/skill-creator/scripts/utils.py b/.opencode/skills/skill-creator/scripts/utils.pynew file mode 100644index 0000000..51b6a07--- /dev/null+++ b/.opencode/skills/skill-creator/scripts/utils.py@@ -0,0 +1,47 @@+"""Shared utilities for skill-creator scripts."""++from pathlib import Path++++def parse_skill_md(skill_path: Path) -> tuple[str, str, str]:+ """Parse a SKILL.md file, returning (name, description, full_content)."""+ content = (skill_path / "SKILL.md").read_text()+ lines = content.split("\n")++ if lines[0].strip() != "---":+ raise ValueError("SKILL.md missing frontmatter (no opening ---)")++ end_idx = None+ for i, line in enumerate(lines[1:], start=1):+ if line.strip() == "---":+ end_idx = i+ break++ if end_idx is None:+ raise ValueError("SKILL.md missing frontmatter (no closing ---)")++ name = ""+ description = ""+ frontmatter_lines = lines[1:end_idx]+ i = 0+ while i < len(frontmatter_lines):+ line = frontmatter_lines[i]+ if line.startswith("name:"):+ name = line[len("name:"):].strip().strip('"').strip("'")+ elif line.startswith("description:"):+ value = line[len("description:"):].strip()+ # Handle YAML multiline indicators (>, |, >-, |-)+ if value in (">", "|", ">-", "|-"):+ continuation_lines: list[str] = []+ i += 1+ while i < len(frontmatter_lines) and (frontmatter_lines[i].startswith(" ") or frontmatter_lines[i].startswith("\t")):+ continuation_lines.append(frontmatter_lines[i].strip())+ i += 1+ description = " ".join(continuation_lines)+ continue+ else:+ description = value.strip('"').strip("'")+ i += 1++ return name, description, content