Harness engineering beginner tutorial, from 0 to 1
TypeScript agentagenticagentic-aiaiai-agentai-agentsdshdsh-pluginharnessharness-engineeringharness-frameworkllm
Improve harness-creator skill
sanbuphy724123cd4fb0
diff --git a/skills/README-CN.md b/skills/README-CN.mdindex b6a9ee0..60851dd 100644--- a/skills/README-CN.md+++ b/skills/README-CN.md@@ -21,10 +21,10 @@面向 AI 编程智能体的生产级 harness 工程技能。帮助创建、评估和改进 agent harness 文件(AGENTS.md、功能清单、验证工作流、会话连续性机制)。-- **5 个参考模式**:记忆持久化、上下文工程、工具注册、多智能体协调、生命周期与引导-- **模板**:AGENTS.md、feature-list.json、init.sh、progress.md-- **5 个内置评估测试用例**-- **双语支持**:English + 中文+- **7 个参考模式**:记忆持久化、技能运行时、上下文工程、工具注册、多智能体协调、生命周期与引导、陷阱+- **模板**:AGENTS.md/CLAUDE.md、feature-list.json、init.sh、progress.md、session-handoff.md+- **脚本**:脚手架生成、校验、HTML 评估报告、结构化 benchmark+- **10 个内置评估测试用例**详见 [harness-creator/README.md](harness-creator/README.md)。@@ -49,10 +49,12 @@ skills/├── README-VI.md # 越南文说明├── README-DE.md # 德文说明└── harness-creator/ # Harness 工程技能- ├── SKILL.md # 主技能定义(双语)+ ├── SKILL.md # 主技能定义├── SKILL.md.en # 英文版├── README.md # 详细文档├── metadata.json # 技能元数据与触发词+ ├── agents/ # 技能 UI 元数据+ ├── scripts/ # 脚手架、校验和 benchmark 辅助脚本├── evals/ # 测试用例├── templates/ # 脚手架模板└── references/ # 深入的模式参考文档@@ -75,7 +77,8 @@ skills/- 无后门、隐藏 URL 或编码载荷- 无数据泄露或硬编码凭证- 无命令注入漏洞-- `init.sh` 仅运行标准 npm 生命周期命令+- 脚本仅使用 Node.js 内置模块+- 生成的 `init.sh` 运行检测到的项目验证命令## 许可证diff --git a/skills/README.md b/skills/README.mdindex efba19c..f28ad8a 100644--- a/skills/README.md+++ b/skills/README.md@@ -21,10 +21,10 @@ This directory contains reusable AI agent skills for the Learn Harness EngineeriProduction harness engineering skill for AI coding agents. Helps create, assess, and improve agent harness files (AGENTS.md, feature lists, verification workflows, session continuity mechanisms).-- **5 reference patterns**: Memory Persistence, Context Engineering, Tool Registry, Multi-Agent Coordination, Lifecycle & Bootstrap-- **Templates**: AGENTS.md, feature-list.json, init.sh, progress.md-- **5 built-in eval test cases**-- **Bilingual**: English + 中文+- **7 reference patterns**: Memory Persistence, Skill Runtime, Context Engineering, Tool Registry, Multi-Agent Coordination, Lifecycle & Bootstrap, Gotchas+- **Templates**: AGENTS.md/CLAUDE.md, feature-list.json, init.sh, progress.md, session-handoff.md+- **Scripts**: scaffold, validate, render HTML assessment, run structural benchmark+- **10 built-in eval test cases**See [harness-creator/README.md](harness-creator/README.md) for full documentation.@@ -49,10 +49,12 @@ skills/├── README-VI.md # Vietnamese version├── README-DE.md # German version└── harness-creator/ # Harness engineering skill- ├── SKILL.md # Main skill definition (bilingual)+ ├── SKILL.md # Main skill definition├── SKILL.md.en # English-only version├── README.md # Detailed documentation├── metadata.json # Skill metadata & triggers+ ├── agents/ # Skill UI metadata+ ├── scripts/ # Scaffold, validate, benchmark helpers├── evals/ # Test cases├── templates/ # Scaffold templates└── references/ # Deep-dive pattern docs@@ -75,7 +77,8 @@ All files in this directory have been audited for security:- No backdoors, hidden URLs, or encoded payloads- No data exfiltration or hardcoded credentials- No command injection vulnerabilities-- `init.sh` runs only standard npm lifecycle commands+- Scripts use only Node.js built-in modules+- Generated `init.sh` runs detected project verification commands## Licensediff --git a/skills/harness-creator/README.md b/skills/harness-creator/README.mdindex bced2f2..8627663 100644--- a/skills/harness-creator/README.md+++ b/skills/harness-creator/README.md@@ -1,206 +1,83 @@-<p align="center">- <a href="../README.md"><img alt="English" src="https://img.shields.io/badge/English-d9d9d9"></a>- <a href="../README-CN.md"><img alt="简体中文" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>- <a href="../README-ZH-TW.md"><img alt="繁體中文" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>- <a href="../README-JA.md"><img alt="日本語" src="https://img.shields.io/badge/日本語-d9d9d9"></a>- <a href="../README-ES.md"><img alt="Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>- <a href="../README-FR.md"><img alt="Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>- <a href="../README-KO.md"><img alt="한국어" src="https://img.shields.io/badge/한국어-d9d9d9"></a>- <a href="../README-AR.md"><img alt="العربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>- <a href="../README-VI.md"><img alt="Tiếng_Việt" src="https://img.shields.io/badge/Tiếng_Việt-d9d9d9"></a>- <a href="../README-DE.md"><img alt="Deutsch" src="https://img.shields.io/badge/Deutsch-d9d9d9"></a>-</p>+# harness-creator-# harness-creator Skill+A compact skill for building and auditing harnesses around AI coding agents.-Production harness engineering skill for AI coding agents, distilled from Learn Harness Engineering course and industry best practices.+It helps a repository provide five things agents need: instructions, state, verification, scope boundaries, and lifecycle handoff.-## Installation+## Install```bashnpx skills add walkinglabs/learn-harness-engineering --skill harness-creator```-Or manually copy the `skills/harness-creator/` directory to your skill path.+Or copy `skills/harness-creator/` into your skill path.-## What This Skill Does+## Use-This skill helps you:-- **Create harnesses from scratch** — AGENTS.md, feature lists, verification workflows-- **Improve existing harnesses** — Five-subsystem assessment with prioritized improvements-- **Design session continuity** — Memory persistence, progress tracking, handoff procedures-- **Benchmark effectiveness** — Before/after comparison with quantitative metrics-- **Apply production patterns** — Memory, context engineering, tool safety, multi-agent coordination--## Core Framework: Five Subsystems--Every harness consists of five subsystems:--1. **Instructions** — AGENTS.md as routing layer, progressive disclosure via docs/ hierarchy-2. **State** — feature_list.json, progress.md, session handoff files-3. **Verification** — Explicit commands that agent MUST run before claiming done-4. **Scope** — One-feature-at-a-time policy, clear definition of done-5. **Lifecycle** — init.sh, clean-state checklists, session continuity mechanisms--## Reference Patterns--This skill includes 6 deep-dive reference documents:--| Pattern | When to Use |-|---------|-------------|-| [Memory Persistence](references/memory-persistence-pattern.md) | Agent forgets between sessions, need persistent project knowledge |-| [Context Engineering](references/context-engineering-pattern.md) | Context budget management, JIT loading, delegation isolation |-| [Tool Registry](references/tool-registry-pattern.md) | Tool safety, concurrency control, permission pipelines |-| [Multi-Agent Coordination](references/multi-agent-pattern.md) | Parallelism, specialization, researcher→implementer workflows |-| [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) | Hooks, background tasks, initialization sequences |-| [Gotchas](references/gotchas.md) | 15 non-obvious failure modes with fixes |--## Usage Examples--### Create Minimal Harness--```-User: "I need to set up AGENTS.md for my TypeScript project"--Skill will:-1. Ask about project context (stack, size, agent tool)-2. Generate AGENTS.md with startup workflow and working rules-3. Create feature_list.json template with placeholder features-4. Create init.sh with verification commands-5. Explain how to use each file-```--### Assess Existing Harness--```-User: "My agent still breaks things even with AGENTS.md"--Skill will:-1. Request current AGENTS.md content-2. Score each of 5 subsystems (1-5 scale)-3. Identify the lowest-scoring subsystem as the highest-priority improvement area-4. Provide prioritized improvement plan with concrete steps-```--### Design Session Continuity--```-User: "Agent forgets everything between sessions"--Skill will:-1. Explain memory layers (instruction vs auto-memory)-2. Design progress.md template for session tracking-3. Create session-handoff.md structure-4. Implement two-step save invariant (topic file → index)+```bash+node skills/harness-creator/scripts/create-harness.mjs --target /path/to/project+node skills/harness-creator/scripts/validate-harness.mjs --target /path/to/project+node skills/harness-creator/scripts/run-benchmark.mjs --target /path/to/project --html /path/to/report.html```-## When to Trigger+The scripts use only Node.js built-in modules. They can be run after copying the skill directory into another repository.-This skill triggers on:-- "Create AGENTS.md / CLAUDE.md"-- "Improve agent reliability"-- "Agent forgets between sessions"-- "Multi-session continuity needed"-- "Benchmark harness effectiveness"-- "Design verification workflow"-- "Memory persistence patterns"-- "Context engineering for agents"+## What It Creates-## When NOT to Use+- `AGENTS.md` or `CLAUDE.md`+- `feature_list.json`+- `progress.md`+- `init.sh`+- `session-handoff.md`-This skill does NOT cover:-- Prompt engineering or system prompt design-- Model selection or fine-tuning-- Generic software architecture-- LLM API integration basics+`create-harness.mjs` detects common project types and package managers. It supports Node/npm/pnpm/yarn/bun, Python, Go, Rust, Maven, Gradle, and .NET at a basic verification-command level.-## Templates Included+## What It Checks-- `templates/agents.md` — AGENTS.md scaffold with working rules-- `templates/feature-list.json` — JSON Schema + example-- `templates/init.sh` — Standard initialization script-- `templates/progress.md` — Session progress log template-- `templates/session-handoff.md` — Handoff structure+`validate-harness.mjs` scores the five harness subsystems:-## Evaluation Framework+1. Instructions+2. State+3. Verification+4. Scope+5. Lifecycle-5 test cases in `evals/evals.json`:-1. **Minimal Harness Creation** — Full setup from scratch-2. **Session Continuity Setup** — Memory and handoff design-3. **Harness Assessment** — Five-subsystem scoring-4. **Verification Workflow Design** — Force agent to verify before done-5. **Memory Taxonomy Design** — What to save vs skip+The score is structural. It tells you whether the harness is present and coherent; it does not replace real before/after agent-session testing.-Run evaluation with skill-creator framework for quantitative benchmarks.+## Status-## Compatibility+- [x] Minimal harness scaffolding+- [x] Five-subsystem validation+- [x] HTML assessment report+- [x] Structural benchmark report+- [x] 10 eval cases+- [x] Generic verification detection for common stacks+- [ ] Optional real before/after agent-session replay-- **Agents**: Claude Code, Codex, Cursor, Windsurf, generic-- **License**: MIT-- **Languages**: English / 中文 (bilingual support in SKILL.md)+## Files-## 兼容性--- **代理工具**: Claude Code, Codex, Cursor, Windsurf, generic-- **许可证**: MIT-- **语言**: 英文 / 中文 (SKILL.md 中双语支持)--## Project Structure--```+```textharness-creator/-├── SKILL.md # Main skill definition-├── metadata.json # Skill metadata, triggers, compatibility-├── evals/-│ └── evals.json # 5 test cases with expectations+├── SKILL.md+├── metadata.json+├── agents/openai.yaml+├── scripts/+│ ├── create-harness.mjs+│ ├── validate-harness.mjs+│ ├── render-assessment-html.mjs+│ ├── run-benchmark.mjs+│ └── lib/harness-utils.mjs├── templates/-│ ├── agents.md # AGENTS.md template-│ ├── feature-list.json # Feature tracker template-│ ├── init.sh # Initialization script-│ └── progress.md # Session progress template-└── references/- ├── memory-persistence-pattern.md- ├── context-engineering-pattern.md- ├── tool-registry-pattern.md- ├── multi-agent-pattern.md- ├── lifecycle-bootstrap-pattern.md- └── gotchas.md # 15 failure modes+│ ├── agents.md+│ ├── feature-list.json+│ ├── feature-list.schema.json+│ ├── init.sh+│ ├── progress.md+│ └── session-handoff.md+├── references/+└── evals/evals.json```-## Development Roadmap--- [x] Chinese translation in SKILL.md (bilingual support added)-- [ ] Python scripts for automated harness generation-- [ ] HTML viewer for harness assessment results-- [ ] Expanded eval set (10+ test cases)-- [ ] Integration with skill-creator benchmark framework-- [ ] Full Chinese localization (harness-creator-zh directory)--## 开发路线图--- [x] SKILL.md 中文翻译(已添加双语支持)-- [ ] Harness 自动生成 Python 脚本-- [ ] Harness 评估结果 HTML 查看器-- [ ] 扩展测试用例(10+ 个)-- [ ] 集成 skill-creator 基准测试框架-- [ ] 完整中文本地化(harness-creator-zh 目录)--## Contributing--Issues and PRs welcome. Key areas for contribution:-- Additional reference patterns (skill runtime, hook lifecycle, etc.)-- More eval test cases covering edge cases-- Script automation for common harness tasks-- Case studies from production deployments--## License--MIT — See LICENSE file for details.--## Acknowledgments+## Boundaries-This skill synthesizes:-- Learn Harness Engineering course framework-- OpenAI Harness Engineering principles-- Anthropic effective harnesses research-- Agentic Harness Patterns skill (pattern extraction methodology)+This skill is for harness engineering, not model selection, prompt tuning alone, or app architecture. Keep project-specific facts in the target repository.diff --git a/skills/harness-creator/SKILL.md b/skills/harness-creator/SKILL.mdindex 91d7d61..802fed8 100644--- a/skills/harness-creator/SKILL.md+++ b/skills/harness-creator/SKILL.md@@ -1,440 +1,107 @@---name: harness-creatordescription: >-- Harness engineering for AI coding agents — five subsystems, memory persistence,- session continuity, verification workflows, scope control, lifecycle management.-when_to_use: >-- Use whenever: building harness from scratch, improving agent reliability, agent forgets- between sessions, agent overreach or scope creep, broken tests after agent work,- multi-session continuity needed, verification gaps, audit harness quality, benchmark- effectiveness, create AGENTS.md/CLAUDE.md, design feature tracking, session handoff.+ Build, audit, and improve lightweight harnesses for AI coding agents: AGENTS.md/CLAUDE.md,+ feature state, verification workflows, scope boundaries, lifecycle handoff,+ memory persistence, context control, tool safety, and multi-agent coordination.license: MIT---# Harness Creator-Production harness engineering for AI coding agents.+Use this skill to make a repository easier for coding agents to start, stay in scope, verify work, and resume across sessions. Keep the harness small enough that agents actually follow it.-**For:** Engineers building or extending coding-agent runtimes, custom agents, multi-session workflows, or anyone who wants their agent to work reliably across sessions.+Not for model selection, prompt tuning in isolation, chat UI design, or general app architecture.-**Not for:** Prompt engineering, model selection, generic software architecture, or one-off agent tasks.+## Core Model-All principles are grounded in the Learn Harness Engineering framework and production agent runtime decisions.+Every useful coding-agent harness has five subsystems:------# Harness Creator(中文版)--面向 AI 编程代理的生产级 Harness 工程技能。--**适用人群:** 构建或扩展编程代理运行时、自定义代理、多会话工作流的工程师,或任何希望代理跨会话可靠工作的人。--**不适用场景:** 提示工程、模型选择、通用软件架构或一次性代理任务。--所有原则均基于 Learn Harness Engineering 框架和生产代理运行时决策。-------## Choose Your Problem--| If you want to... | Read |-|---|---|-| Make the agent remember corrections and project rules between sessions | [Memory Persistence](references/memory-persistence-pattern.md) |-| Package reusable workflows and domain knowledge | [Skill Runtime](references/skill-runtime-pattern.md) |-| Let the agent work powerfully but not dangerously | [Tool Registry & Safety](references/tool-registry-pattern.md) |-| Give the agent the right context at the right cost | [Context Engineering](references/context-engineering-pattern.md) |-| Split work across multiple agents without chaos | [Multi-agent Coordination](references/multi-agent-pattern.md) |-| Extend behavior with hooks, background tasks, startup logic | [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) |-| Build the complete 5-subsystem harness | [Five Subsystems Guide](#the-five-subsystem-harness-framework) |--**Before you start building:** Read the [Gotchas](#gotchas) — these are the non-obvious failure modes that cost the most time.-------## 选择你要解决的问题--| 如果你想... | 阅读 |-|---|---|-| 让代理在会话之间记住修正和项目规则 | [记忆持久化](references/memory-persistence-pattern.md) |-| 打包可重复使用的工作流和领域知识 | [技能运行时](references/skill-runtime-pattern.md) |-| 让代理强大但安全地工作 | [工具注册与安全](references/tool-registry-pattern.md) |-| 以合适的成本给代理合适的上下文 | [上下文工程](references/context-engineering-pattern.md) |-| 在多个代理之间分配工作而不混乱 | [多代理协调](references/multi-agent-pattern.md) |-| 使用 hooks、后台任务、启动逻辑扩展行为 | [生命周期与引导](references/lifecycle-bootstrap-pattern.md) |-| 构建完整的 5 子系统 harness | [五子系统指南](#the-five-subsystem-harness-framework) |--**开始构建之前:** 阅读 [陷阱](references/gotchas.md) — 这些是最耗时的非明显失败模式。-------## The Five-Subsystem Harness Framework--**Every harness consists of five subsystems:**--1. **Instructions (Recipe Shelf)**: AGENTS.md, CLAUDE.md, docs/ hierarchy-2. **State (Prep Station)**: feature_list.json, progress.md, session-handoff.md-3. **Verification (Quality Check Window)**: Verification commands, test suites, type checks-4. **Scope (Task Boundaries)**: One-feature-at-a-time policies, definition of done-5. **Lifecycle (Session Management)**: init.sh, clean-state checklists, handoff procedures--**When creating or improving a harness, systematically address each subsystem.**-------## Creating a Harness--### Phase 1: Context Gathering--Start by understanding the user's situation:--1. **What project is this for?** (tech stack, size, complexity)-2. **What agent tool are they using?** (Claude Code, Codex, Cursor, etc.)-3. **What exists already?** (any AGENTS.md, progress tracking, verification?)-4. **What problems are they experiencing?** (agent overreach, lost context, broken tests?)-5. **What's the team's tolerance for structure?** (minimal vs. comprehensive)+| Subsystem | Minimal artifact | Purpose |+|---|---|---|+| Instructions | `AGENTS.md` or `CLAUDE.md` | Startup path, working rules, definition of done |+| State | `feature_list.json`, `progress.md` | Current feature, status, evidence, next step |+| Verification | `init.sh` or documented commands | Tests/checks the agent must run before claiming done |+| Scope | Feature dependencies and done criteria | Prevents overreach and half-finished work |+| Lifecycle | `session-handoff.md`, end-of-session routine | Makes the next session restartable |-If the user hasn't provided this context, ask before proceeding.+## First Move-### Phase 2: Harness Assessment (Existing Projects)+1. Inspect what already exists: instruction files, feature/state files, verification commands, docs, package manifests.+2. Ask only for missing context that cannot be inferred safely: target agent, desired file name, tolerance for structure, and whether overwriting is allowed.+3. Prefer a minimal harness first. Add memory, tool safety, multi-agent, or benchmark details only when the user's problem calls for them.-If the user has an existing harness, assess it using the five-tuple framework:+## Common Tasks-For each subsystem, score 1-5:-- **5**: Exemplary, documented, consistently followed-- **4**: Good, mostly complete, occasional gaps-- **3**: Adequate, covers basics, missing polish-- **2**: Weak, incomplete, inconsistently applied-- **1**: Missing or actively harmful+### Create a harness-Identify the lowest-scoring subsystem as the highest-priority improvement area. Treat it as a candidate bottleneck, then confirm with failure logs, verification evidence, and task outcomes before calling it the bottleneck.--### Phase 3: Design--Based on the assessment, design the harness components:--**Instructions:**-- Create a short AGENTS.md (~50-100 lines) as the routing layer-- Link to detailed docs in docs/ directory (ARCHITECTURE.md, PRODUCT.md, etc.)-- Define startup workflow: what the agent reads before coding--**State:**-- Create feature_list.json with feature definitions and status tracking-- Create or update progress.md for session continuity-- Design session-handoff.md template if needed--**Verification:**-- List explicit verification commands in AGENTS.md-- Ensure init.sh runs verification-- Design quality score tracking if appropriate--**Scope:**-- Define one-feature-at-a-time policy-- Document feature dependencies-- Create definition of done checklist--**Lifecycle:**-- Create init.sh for initialization-- Design clean-state checklist-- Document session handoff procedure--### Phase 4: Implementation--Create the harness files. Use bundled scripts where available:+Use the bundled script when working on a local repository:```bash-# Use bundled scripts from scripts/ directory-# (See scripts/ section for available tools)+node skills/harness-creator/scripts/create-harness.mjs --target /path/to/project```-### Phase 5: Testing and Benchmarking--Test the harness with real agent sessions:--1. **Baseline**: Run a representative task without the harness-2. **With Harness**: Run the same task with the harness-3. **Measure**: Success rate, time, token usage, rework-4. **Compare**: Quantify the improvement--For rigorous benchmarking, see the "Running Benchmarks" section below.-------## Harness File Templates--### AGENTS.md Structure--A minimal AGENTS.md should include:--```markdown-# AGENTS.md--[One-sentence project purpose]--## Startup Workflow--Before writing code:-1. [Step 1: e.g., Read this file]-2. [Step 2: e.g., Read ARCHITECTURE.md]-3. [Step 3: e.g., Run ./init.sh]-4. [Step 4: e.g., Read feature_list.json]+Options:-## Working Rules+- `--agent-file CLAUDE.md` for Claude-oriented projects.+- `--package-manager npm|pnpm|yarn|bun` when detection is wrong.+- `--commands "cmd one,cmd two"` for custom verification.+- `--force` only after confirming overwrites are acceptable.-- [Rule 1: e.g., One feature at a time]-- [Rule 2: e.g., Verification required before claiming done]-- [Rule 3: e.g., Update progress before ending session]+Then explain what was created and how the user should replace placeholder feature entries.-## Required Artifacts+### Audit an existing harness-- `feature_list.json`: Feature state tracker-- `progress.md`: Session continuity log-- `init.sh`: Standard startup and verification--## Definition of Done--A feature is done when:-- [ ] Implementation complete-- [ ] Verification passed-- [ ] Evidence recorded-- [ ] Repository restartable--## End of Session--Before ending:-1. Update progress.md-2. Update feature_list.json-3. Record blockers/risks-4. Commit with descriptive message-5. Leave clean restart path-```--### feature_list.json Structure--```json-{- "features": [- {- "id": "feat-001",- "name": "Document Import",- "description": "Allow users to import PDF and TXT documents",- "dependencies": [],- "status": "done",- "evidence": "tests pass, manual verification on 2024-01-15"- },- {- "id": "feat-002",- "name": "Document Chunking",- "description": "Split documents into ~500 char chunks with metadata",- "dependencies": ["feat-001"],- "status": "in-progress",- "evidence": ""- }- ]-}-```--### init.sh Structure+Run:```bash-#!/bin/bash-set -e--echo "=== Installing dependencies ==="-npm install--echo "=== Running type check ==="-npm run check--echo "=== Running tests ==="-npm test--echo "=== Building application ==="-npm run build--echo "=== Verification complete ==="+node skills/harness-creator/scripts/validate-harness.mjs --target /path/to/project```------## Running Benchmarks--To measure harness effectiveness:--### Step 1: Define Representative Tasks--Pick 2-3 tasks that are:-- Real work the user would actually do-- Challenging enough to fail without proper harness-- Verifiable (clear success criteria)--### Step 2: Run Comparative Sessions--For each task:-- **Without Harness**: Run the task on a clean repo copy-- **With Harness**: Run the same task with the harness in place--Record:-- Success/failure-- Time taken-- Token usage-- Rework required-- Session restarts needed+Report the five subsystem scores, the lowest-scoring area, and the first 2-3 changes that would improve reliability. Treat the lowest score as a candidate bottleneck; confirm with failures, logs, or task outcomes before claiming causality.-### Step 3: Aggregate Results--Calculate:-- Success rate improvement-- Time efficiency change-- Token efficiency change-- Qualitative feedback--### Step 4: Iterate--Use results to identify:-- Which harness components add most value-- Which components are over-engineered-- Where to focus improvement efforts-------## Bundled Resources--### References (Deep-Dive Patterns)--| Document | Covers |-|----------|--------|-| [Memory Persistence](references/memory-persistence-pattern.md) | Four-level instruction hierarchy, auto-memory taxonomy, background extraction |-| [Context Engineering](references/context-engineering-pattern.md) | Select / Compress / Isolate / Write operations, budget management |-| [Tool Registry](references/tool-registry-pattern.md) | Fail-closed registration, per-call concurrency, permission pipeline |-| [Multi-Agent](references/multi-agent-pattern.md) | Coordinator / Fork / Swarm patterns, context sharing |-| [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) | Hook system, long-running tasks, dependency-ordered init |-| [Gotchas](references/gotchas.md) | 15 non-obvious failure modes with fixes |--### Templates--- `templates/agents.md` — AGENTS.md / CLAUDE.md skeleton-- `templates/feature-list.json` — Feature state tracker-- `templates/init.sh` — Standard initialization script-- `templates/progress.md` — Session progress log-- `templates/session-handoff.md` — Session handoff template--### Scripts (Optional)--- `scripts/create-harness.ts` — Generate harness files from templates-- `scripts/validate-harness.ts` — Check harness completeness-- `scripts/run-benchmark.ts` — Execute harness effectiveness comparison-----+### Produce a report-## Gotchas+Use when the user wants a shareable assessment:-Non-obvious principles that will cause bugs if you violate them:--1. **Memory index caps fire silently** — Long entries invisible once cap hit. Keep hooks to one line.-2. **Priority ordering counterintuitive** — Local beats project beats user beats org. Test full stack.-3. **Extraction timing creates race window** — User can start next turn before background extraction completes.-4. **Derivable content doesn't belong in memory** — Architecture and code patterns are in the repo already.-5. **Concurrent classification is per-call, not per-tool** — Same tool safe for some inputs, unsafe for others.-6. **Permission evaluation has side effects** — Tracks denials, transforms modes, updates state.-7. **Most async work skips "pending" state** — Work units register directly as "running".-8. **Fork children must not fork** — Recursive guard preserves single-level invariant.-9. **Context builders memoized but manually invalidated** — Add invalidation or face staleness.-10. **Hook trust all-or-nothing** — One untrusted hook disables entire extension system.-11. **Eviction requires notification** — Terminal work unit only GC-eligible after parent notified.-12. **Skill listing budgets tight** — Front-load distinctive trigger language, tails get cut.--**Full guide**: [Gotchas](references/gotchas.md) — 15 failure modes with fixes.--## 陷阱(Gotchas)--违反这些非明显原则会导致 bug:--1. **记忆索引上限静默触发** — 条目过长超上限后不可见。保持钩子单行。-2. **优先级顺序反直觉** — 本地胜过项目胜过用户胜过组织。测试完整栈。-3. **提取时序产生竞争窗口** — 用户可在后台提取完成前开始下一轮。-4. **可推导内容不应存入记忆** — 架构和代码模式已在仓库中。-5. **并发分类按调用而非按工具** — 同一工具对某些输入安全,对其他不安全。-6. **权限评估有副作用** — 跟踪拒绝、转换模式、更新状态。-7. **大多数异步工作跳过"pending"状态** — 工作单元直接注册为"运行中"。-8. **Fork 子节点不能 Fork** — 递归防护保持单层不变量。-9. **上下文构建器缓存但手动失效** — 添加失效或面对过时。-10. **Hook 信任全有或全无** — 一个不可信 hook 禁用整个扩展系统。-11. **驱逐需要通知** — 终端工作单元仅在父节点通知后可 GC。-12. **Skill 列表预算紧张** — 前置独特触发语言,尾部被截断。--**完整指南**:[陷阱](references/gotchas.md) — 15 种失败模式及修复方法。-------## When to Use This Skill--Use this skill when:--- User says "I need to set up AGENTS.md for my project"-- User wants to improve their agent's reliability-- User is experiencing agent failures, lost context, or broken work-- User asks "how do I make my agent work better?"-- User wants to benchmark harness effectiveness-- User needs templates for harness files-- User is following the Learn Harness Engineering course-------## Communication Style--- Explain harness concepts in practical terms (kitchen analogy works well)-- Focus on measurable outcomes, not theoretical perfection-- Start minimal, add structure as needed-- Show before/after comparisons to build confidence-- Acknowledge tradeoffs (more structure = more reliability but more upfront work)-------## Getting Started--### If the user is new to harness engineering:--1. **Start with assessment**: Run the five-tuple assessment on their current setup-2. **Pick lowest-scoring subsystem**: Focus improvement efforts there first-3. **Create minimal viable harness**: AGENTS.md + init.sh + feature_list.json-4. **Test with real task**: Measure before/after improvement--### If the user is experienced:--1. **Ask what specific problem**: Don't assume — let them describe the pain point-2. **Understand harness maturity**: What exists already? What's working?-3. **Design targeted improvements**: Use reference patterns for guidance-4. **Optionally run benchmarks**: Quantify impact with before/after comparison+```bash+node skills/harness-creator/scripts/render-assessment-html.mjs --target /path/to/project+node skills/harness-creator/scripts/run-benchmark.mjs --target /path/to/project --html /path/to/report.html+```----+Be clear that this is a structural benchmark. Real effectiveness still needs before/after agent sessions on representative tasks.-## When NOT to Use This Skill+## When to Read References-This skill is about the **harness** around an agent, not:-- Prompt engineering or system prompt design-- Model selection or fine-tuning-- Generic software architecture (MVC, microservices)-- Chat UIs or conversational interfaces-- LLM API integration basics+Load only the reference needed for the user's problem:-If your question is about the model itself rather than the system around it, this skill does not apply.+- Memory across sessions: [Memory Persistence](references/memory-persistence-pattern.md)+- Reusable workflows as skills: [Skill Runtime](references/skill-runtime-pattern.md)+- Permissions, tools, concurrency: [Tool Registry & Safety](references/tool-registry-pattern.md)+- Context budget and progressive disclosure: [Context Engineering](references/context-engineering-pattern.md)+- Delegation and parallel agents: [Multi-Agent Coordination](references/multi-agent-pattern.md)+- Hooks, startup, long-running work: [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md)+- Non-obvious failure modes: [Gotchas](references/gotchas.md)----+## Design Rules-## Further Resources+- Keep the root instruction file short: routing and invariants, not a full manual.+- Put project facts in project docs, not in the skill.+- Make verification commands explicit and runnable.+- Require evidence before marking a feature done.+- Use one active feature unless the harness has explicit multi-agent ownership boundaries.+- Prefer append/update state files over relying on chat history.+- Never hide destructive behavior in scripts; overwrites require explicit user approval.-- [Learn Harness Engineering Documentation](https://walkinglabs.github.io/learn-harness-engineering/)-- [OpenAI: Harness Engineering](https://openai.com/index/harness-engineering/)-- [Anthropic: Effective Harnesses for Long-Running Agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents/)-- [Anthropic: Harness Design for Long-Running Apps](https://www.anthropic.com/engineering/harness-design-long-running-apps)-- [Awesome Harness Engineering](https://github.com/walkinglabs/awesome-harness-engineering)-- [Agentic Harness Patterns Skill](https://github.com/keli-wen/agentic-harness-patterns-skill) — Reference implementation for pattern extraction+## Deliverable Checklist----+For a usable minimal harness, leave the target project with:-## Further Resources+- [ ] `AGENTS.md` or `CLAUDE.md`+- [ ] `feature_list.json`+- [ ] `progress.md`+- [ ] `init.sh`+- [ ] Optional `session-handoff.md` for multi-session work+- [ ] Documented verification evidence or next action-- [Learn Harness Engineering Documentation](https://walkinglabs.github.io/learn-harness-engineering/)-- [OpenAI: Harness Engineering](https://openai.com/index/harness-engineering/)-- [Anthropic: Effective Harnesses for Long-Running Agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents/)-- [Awesome Harness Engineering](https://github.com/walkinglabs/awesome-harness-engineering)+If you cannot create files, provide exact file contents and commands instead.diff --git a/skills/harness-creator/SKILL.md.en b/skills/harness-creator/SKILL.md.enindex 131be49..802fed8 100644--- a/skills/harness-creator/SKILL.md.en+++ b/skills/harness-creator/SKILL.md.en@@ -1,393 +1,107 @@---name: harness-creatordescription: >-- Harness engineering for AI coding agents — five subsystems, memory persistence,- session continuity, verification workflows, scope control, lifecycle management.-when_to_use: >-- Use whenever: building harness from scratch, improving agent reliability, agent forgets- between sessions, agent overreach or scope creep, broken tests after agent work,- multi-session continuity needed, verification gaps, audit harness quality, benchmark- effectiveness, create AGENTS.md/CLAUDE.md, design feature tracking, session handoff.+ Build, audit, and improve lightweight harnesses for AI coding agents: AGENTS.md/CLAUDE.md,+ feature state, verification workflows, scope boundaries, lifecycle handoff,+ memory persistence, context control, tool safety, and multi-agent coordination.license: MIT---# Harness Creator-Production harness engineering for AI coding agents.+Use this skill to make a repository easier for coding agents to start, stay in scope, verify work, and resume across sessions. Keep the harness small enough that agents actually follow it.-**For:** Engineers building or extending coding-agent runtimes, custom agents, multi-session workflows, or anyone who wants their agent to work reliably across sessions.+Not for model selection, prompt tuning in isolation, chat UI design, or general app architecture.-**Not for:** Prompt engineering, model selection, generic software architecture, or one-off agent tasks.+## Core Model-All principles are grounded in the Learn Harness Engineering framework and production agent runtime decisions.+Every useful coding-agent harness has five subsystems:------## Choose Your Problem--| If you want to... | Read |-|---|---|-| Make the agent remember corrections and project rules between sessions | [Memory Persistence](references/memory-persistence-pattern.md) |-| Package reusable workflows and domain knowledge | [Skill Runtime](references/skill-runtime-pattern.md) |-| Let the agent work powerfully but not dangerously | [Tool Registry & Safety](references/tool-registry-pattern.md) |-| Give the agent the right context at the right cost | [Context Engineering](references/context-engineering-pattern.md) |-| Split work across multiple agents without chaos | [Multi-agent Coordination](references/multi-agent-pattern.md) |-| Extend behavior with hooks, background tasks, startup logic | [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) |-| Build the complete 5-subsystem harness | [Five Subsystems Guide](#the-five-subsystem-harness-framework) |--**Before you start building:** Read the [Gotchas](#gotchas) — these are the non-obvious failure modes that cost the most time.-------## The Five-Subsystem Harness Framework--**Every harness consists of five subsystems:**--1. **Instructions (Recipe Shelf)**: AGENTS.md, CLAUDE.md, docs/ hierarchy-2. **State (Prep Station)**: feature_list.json, progress.md, session-handoff.md-3. **Verification (Quality Check Window)**: Verification commands, test suites, type checks-4. **Scope (Task Boundaries)**: One-feature-at-a-time policies, definition of done-5. **Lifecycle (Session Management)**: init.sh, clean-state checklists, handoff procedures--**When creating or improving a harness, systematically address each subsystem.**-------## Creating a Harness--### Phase 1: Context Gathering--Start by understanding the user's situation:--1. **What project is this for?** (tech stack, size, complexity)-2. **What agent tool are they using?** (Claude Code, Codex, Cursor, etc.)-3. **What exists already?** (any AGENTS.md, progress tracking, verification?)-4. **What problems are they experiencing?** (agent overreach, lost context, broken tests?)-5. **What's the team's tolerance for structure?** (minimal vs. comprehensive)--If the user hasn't provided this context, ask before proceeding.--### Phase 2: Harness Assessment (Existing Projects)--If the user has an existing harness, assess it using the five-tuple framework:--For each subsystem, score 1-5:-- **5**: Exemplary, documented, consistently followed-- **4**: Good, mostly complete, occasional gaps-- **3**: Adequate, covers basics, missing polish-- **2**: Weak, incomplete, inconsistently applied-- **1**: Missing or actively harmful--Identify the lowest-scoring subsystem as the highest-priority improvement area. Treat it as a candidate bottleneck, then confirm with failure logs, verification evidence, and task outcomes before calling it the bottleneck.--### Phase 3: Design--Based on the assessment, design the harness components:--**Instructions:**-- Create a short AGENTS.md (~50-100 lines) as the routing layer-- Link to detailed docs in docs/ directory (ARCHITECTURE.md, PRODUCT.md, etc.)-- Define startup workflow: what the agent reads before coding+| Subsystem | Minimal artifact | Purpose |+|---|---|---|+| Instructions | `AGENTS.md` or `CLAUDE.md` | Startup path, working rules, definition of done |+| State | `feature_list.json`, `progress.md` | Current feature, status, evidence, next step |+| Verification | `init.sh` or documented commands | Tests/checks the agent must run before claiming done |+| Scope | Feature dependencies and done criteria | Prevents overreach and half-finished work |+| Lifecycle | `session-handoff.md`, end-of-session routine | Makes the next session restartable |-**State:**-- Create feature_list.json with feature definitions and status tracking-- Create or update progress.md for session continuity-- Design session-handoff.md template if needed+## First Move-**Verification:**-- List explicit verification commands in AGENTS.md-- Ensure init.sh runs verification-- Design quality score tracking if appropriate+1. Inspect what already exists: instruction files, feature/state files, verification commands, docs, package manifests.+2. Ask only for missing context that cannot be inferred safely: target agent, desired file name, tolerance for structure, and whether overwriting is allowed.+3. Prefer a minimal harness first. Add memory, tool safety, multi-agent, or benchmark details only when the user's problem calls for them.-**Scope:**-- Define one-feature-at-a-time policy-- Document feature dependencies-- Create definition of done checklist+## Common Tasks-**Lifecycle:**-- Create init.sh for initialization-- Design clean-state checklist-- Document session handoff procedure+### Create a harness-### Phase 4: Implementation--Create the harness files. Use bundled scripts where available:+Use the bundled script when working on a local repository:```bash-# Use bundled scripts from scripts/ directory-# (See scripts/ section for available tools)+node skills/harness-creator/scripts/create-harness.mjs --target /path/to/project```-### Phase 5: Testing and Benchmarking--Test the harness with real agent sessions:--1. **Baseline**: Run a representative task without the harness-2. **With Harness**: Run the same task with the harness-3. **Measure**: Success rate, time, token usage, rework-4. **Compare**: Quantify the improvement--For rigorous benchmarking, see the "Running Benchmarks" section below.-------## Harness File Templates--### AGENTS.md Structure--A minimal AGENTS.md should include:--```markdown-# AGENTS.md--[One-sentence project purpose]--## Startup Workflow+Options:-Before writing code:-1. [Step 1: e.g., Read this file]-2. [Step 2: e.g., Read ARCHITECTURE.md]-3. [Step 3: e.g., Run ./init.sh]-4. [Step 4: e.g., Read feature_list.json]+- `--agent-file CLAUDE.md` for Claude-oriented projects.+- `--package-manager npm|pnpm|yarn|bun` when detection is wrong.+- `--commands "cmd one,cmd two"` for custom verification.+- `--force` only after confirming overwrites are acceptable.-## Working Rules+Then explain what was created and how the user should replace placeholder feature entries.-- [Rule 1: e.g., One feature at a time]-- [Rule 2: e.g., Verification required before claiming done]-- [Rule 3: e.g., Update progress before ending session]+### Audit an existing harness-## Required Artifacts--- `feature_list.json`: Feature state tracker-- `progress.md`: Session continuity log-- `init.sh`: Standard startup and verification--## Definition of Done--A feature is done when:-- [ ] Implementation complete-- [ ] Verification passed-- [ ] Evidence recorded-- [ ] Repository restartable--## End of Session--Before ending:-1. Update progress.md-2. Update feature_list.json-3. Record blockers/risks-4. Commit with descriptive message-5. Leave clean restart path-```--### feature_list.json Structure--```json-{- "features": [- {- "id": "feat-001",- "name": "Document Import",- "description": "Allow users to import PDF and TXT documents",- "dependencies": [],- "status": "done",- "evidence": "tests pass, manual verification on 2024-01-15"- },- {- "id": "feat-002",- "name": "Document Chunking",- "description": "Split documents into ~500 char chunks with metadata",- "dependencies": ["feat-001"],- "status": "in-progress",- "evidence": ""- }- ]-}-```--### init.sh Structure+Run:```bash-#!/bin/bash-set -e--echo "=== Installing dependencies ==="-npm install--echo "=== Running type check ==="-npm run check--echo "=== Running tests ==="-npm test--echo "=== Building application ==="-npm run build--echo "=== Verification complete ==="+node skills/harness-creator/scripts/validate-harness.mjs --target /path/to/project```------## Running Benchmarks--To measure harness effectiveness:--### Step 1: Define Representative Tasks--Pick 2-3 tasks that are:-- Real work the user would actually do-- Challenging enough to fail without proper harness-- Verifiable (clear success criteria)--### Step 2: Run Comparative Sessions--For each task:-- **Without Harness**: Run the task on a clean repo copy-- **With Harness**: Run the same task with the harness in place--Record:-- Success/failure-- Time taken-- Token usage-- Rework required-- Session restarts needed--### Step 3: Aggregate Results--Calculate:-- Success rate improvement-- Time efficiency change-- Token efficiency change-- Qualitative feedback+Report the five subsystem scores, the lowest-scoring area, and the first 2-3 changes that would improve reliability. Treat the lowest score as a candidate bottleneck; confirm with failures, logs, or task outcomes before claiming causality.-### Step 4: Iterate--Use results to identify:-- Which harness components add most value-- Which components are over-engineered-- Where to focus improvement efforts-------## Bundled Resources--### References (Deep-Dive Patterns)--| Document | Covers |-|----------|--------|-| [Memory Persistence](references/memory-persistence-pattern.md) | Four-level instruction hierarchy, auto-memory taxonomy, background extraction |-| [Context Engineering](references/context-engineering-pattern.md) | Select / Compress / Isolate / Write operations, budget management |-| [Tool Registry](references/tool-registry-pattern.md) | Fail-closed registration, per-call concurrency, permission pipeline |-| [Multi-Agent](references/multi-agent-pattern.md) | Coordinator / Fork / Swarm patterns, context sharing |-| [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) | Hook system, long-running tasks, dependency-ordered init |-| [Gotchas](references/gotchas.md) | 15 non-obvious failure modes with fixes |--### Templates--- `templates/agents.md` — AGENTS.md / CLAUDE.md skeleton-- `templates/feature-list.json` — Feature state tracker-- `templates/init.sh` — Standard initialization script-- `templates/progress.md` — Session progress log-- `templates/session-handoff.md` — Session handoff template--### Scripts (Optional)--- `scripts/create-harness.ts` — Generate harness files from templates-- `scripts/validate-harness.ts` — Check harness completeness-- `scripts/run-benchmark.ts` — Execute harness effectiveness comparison-----+### Produce a report-## Gotchas+Use when the user wants a shareable assessment:-Non-obvious principles that will cause bugs if you violate them:--1. **Memory index caps fire silently** — Long entries invisible once cap hit. Keep hooks to one line.-2. **Priority ordering counterintuitive** — Local beats project beats user beats org. Test full stack.-3. **Extraction timing creates race window** — User can start next turn before background extraction completes.-4. **Derivable content doesn't belong in memory** — Architecture and code patterns are in the repo already.-5. **Concurrent classification is per-call, not per-tool** — Same tool safe for some inputs, unsafe for others.-6. **Permission evaluation has side effects** — Tracks denials, transforms modes, updates state.-7. **Most async work skips "pending" state** — Work units register directly as "running".-8. **Fork children must not fork** — Recursive guard preserves single-level invariant.-9. **Context builders memoized but manually invalidated** — Add invalidation or face staleness.-10. **Hook trust all-or-nothing** — One untrusted hook disables entire extension system.-11. **Eviction requires notification** — Terminal work unit only GC-eligible after parent notified.-12. **Skill listing budgets tight** — Front-load distinctive trigger language, tails get cut.--**Full guide**: [Gotchas](references/gotchas.md) — 15 failure modes with fixes.-------## When to Use This Skill--Use this skill when:--- User says "I need to set up AGENTS.md for my project"-- User wants to improve their agent's reliability-- User is experiencing agent failures, lost context, or broken work-- User asks "how do I make my agent work better?"-- User wants to benchmark harness effectiveness-- User needs templates for harness files-- User is following the Learn Harness Engineering course-------## Communication Style--- Explain harness concepts in practical terms (kitchen analogy works well)-- Focus on measurable outcomes, not theoretical perfection-- Start minimal, add structure as needed-- Show before/after comparisons to build confidence-- Acknowledge tradeoffs (more structure = more reliability but more upfront work)-------## Getting Started--### If the user is new to harness engineering:--1. **Start with assessment**: Run the five-tuple assessment on their current setup-2. **Pick lowest-scoring subsystem**: Focus improvement efforts there first-3. **Create minimal viable harness**: AGENTS.md + init.sh + feature_list.json-4. **Test with real task**: Measure before/after improvement--### If the user is experienced:--1. **Ask what specific problem**: Don't assume — let them describe the pain point-2. **Understand harness maturity**: What exists already? What's working?-3. **Design targeted improvements**: Use reference patterns for guidance-4. **Optionally run benchmarks**: Quantify impact with before/after comparison+```bash+node skills/harness-creator/scripts/render-assessment-html.mjs --target /path/to/project+node skills/harness-creator/scripts/run-benchmark.mjs --target /path/to/project --html /path/to/report.html+```----+Be clear that this is a structural benchmark. Real effectiveness still needs before/after agent sessions on representative tasks.-## When NOT to Use This Skill+## When to Read References-This skill is about the **harness** around an agent, not:-- Prompt engineering or system prompt design-- Model selection or fine-tuning-- Generic software architecture (MVC, microservices)-- Chat UIs or conversational interfaces-- LLM API integration basics+Load only the reference needed for the user's problem:-If your question is about the model itself rather than the system around it, this skill does not apply.+- Memory across sessions: [Memory Persistence](references/memory-persistence-pattern.md)+- Reusable workflows as skills: [Skill Runtime](references/skill-runtime-pattern.md)+- Permissions, tools, concurrency: [Tool Registry & Safety](references/tool-registry-pattern.md)+- Context budget and progressive disclosure: [Context Engineering](references/context-engineering-pattern.md)+- Delegation and parallel agents: [Multi-Agent Coordination](references/multi-agent-pattern.md)+- Hooks, startup, long-running work: [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md)+- Non-obvious failure modes: [Gotchas](references/gotchas.md)----+## Design Rules-## Further Resources+- Keep the root instruction file short: routing and invariants, not a full manual.+- Put project facts in project docs, not in the skill.+- Make verification commands explicit and runnable.+- Require evidence before marking a feature done.+- Use one active feature unless the harness has explicit multi-agent ownership boundaries.+- Prefer append/update state files over relying on chat history.+- Never hide destructive behavior in scripts; overwrites require explicit user approval.-- [Learn Harness Engineering Documentation](https://walkinglabs.github.io/learn-harness-engineering/)-- [OpenAI: Harness Engineering](https://openai.com/index/harness-engineering/)-- [Anthropic: Effective Harnesses for Long-Running Agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents/)-- [Anthropic: Harness Design for Long-Running Apps](https://www.anthropic.com/engineering/harness-design-long-running-apps)-- [Awesome Harness Engineering](https://github.com/walkinglabs/awesome-harness-engineering)-- [Agentic Harness Patterns Skill](https://github.com/keli-wen/agentic-harness-patterns-skill) — Reference implementation for pattern extraction+## Deliverable Checklist----+For a usable minimal harness, leave the target project with:-## Further Resources+- [ ] `AGENTS.md` or `CLAUDE.md`+- [ ] `feature_list.json`+- [ ] `progress.md`+- [ ] `init.sh`+- [ ] Optional `session-handoff.md` for multi-session work+- [ ] Documented verification evidence or next action-- [Learn Harness Engineering Documentation](https://walkinglabs.github.io/learn-harness-engineering/)-- [OpenAI: Harness Engineering](https://openai.com/index/harness-engineering/)-- [Anthropic: Effective Harnesses for Long-Running Agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents/)-- [Awesome Harness Engineering](https://github.com/walkinglabs/awesome-harness-engineering)+If you cannot create files, provide exact file contents and commands instead.diff --git a/skills/harness-creator/agents/openai.yaml b/skills/harness-creator/agents/openai.yamlnew file mode 100644index 0000000..500d7ae--- /dev/null+++ b/skills/harness-creator/agents/openai.yaml@@ -0,0 +1,7 @@+interface:+ display_name: "Harness Creator"+ short_description: "Scaffold and audit coding-agent harnesses"+ default_prompt: "Use $harness-creator to create, validate, and benchmark a production-ready coding-agent harness for this repository."++policy:+ allow_implicit_invocation: truediff --git a/skills/harness-creator/evals/evals.json b/skills/harness-creator/evals/evals.jsonindex af67269..723d5ee 100644--- a/skills/harness-creator/evals/evals.json+++ b/skills/harness-creator/evals/evals.json@@ -11,7 +11,7 @@"AGENTS.md includes startup workflow (read files, run init, check feature list)","AGENTS.md includes one-feature-at-a-time policy","feature_list.json has valid JSON with id, name, description, status fields",- "init.sh runs npm install, type check, and test commands",+ "init.sh runs install, type/static check, test, and build commands when available","All files are in project root directory"]},@@ -37,7 +37,7 @@"expectations": ["Assessment covers all 5 subsystems: Instructions, State, Verification, Scope, Lifecycle","Each subsystem scored 1-5 with justification",- "Lowest-scoring subsystem identified as bottleneck",+ "Lowest-scoring subsystem identified as bottleneck or candidate bottleneck","Prioritized improvement plan with 2-3 concrete next steps"]},@@ -49,9 +49,9 @@"files": [],"expectations": ["Explicit verification commands listed (tests, lint, type-check, build)",- "AGENTS.md includes 'Definition of Done' section with verification requirement",+ "AGENTS.md includes Definition of Done section with verification requirement","End-of-session checklist includes verification evidence recording",- "Optional: quality score tracking mechanism described"+ "Failure path says not to claim done when verification fails"]},{@@ -66,6 +66,71 @@"Type taxonomy with 3-4 types (e.g., user/feedback/project/reference)","Clear guidance on what NOT to save (derivable content)"]+ },+ {+ "id": 6,+ "name": "Tool Safety Design",+ "prompt": "My custom coding agent can run shell commands and edit files. Design the tool registry and permission harness so it is powerful but does not silently do dangerous things.",+ "expected_output": "Tool registry policy with per-call concurrency classification, default permissions, denial handling, and audit trail",+ "files": [],+ "expectations": [+ "Sensitive tools default to ask or deny instead of allow",+ "Concurrency safety is classified per call rather than per tool",+ "Permission evaluation side effects are acknowledged and not cached unsafely",+ "Audit trail records command, decision, and reason"+ ]+ },+ {+ "id": 7,+ "name": "Context Budget Plan",+ "prompt": "My agent loads too many docs at startup and gets slow. Design a context strategy that keeps important context while controlling token cost.",+ "expected_output": "Progressive disclosure plan using select/write/compress/isolate operations and explicit context budgets",+ "files": [],+ "expectations": [+ "Defines always-loaded metadata versus on-demand references",+ "Uses SELECT, WRITE, COMPRESS, and ISOLATE operations",+ "Includes hard caps or budget thresholds",+ "Explains invalidation for memoized context builders"+ ]+ },+ {+ "id": 8,+ "name": "Multi-Agent Coordination",+ "prompt": "I want multiple agents to work on a large refactor without stepping on each other. Design a multi-agent harness and rules for delegation.",+ "expected_output": "Coordinator/delegation design with ownership boundaries, context sharing policy, and merge/review gates",+ "files": [],+ "expectations": [+ "Defines coordinator, worker, and optional reviewer responsibilities",+ "Assigns disjoint file or module ownership",+ "Prevents recursive fork children from forking again",+ "Includes integration and verification gate before claiming done"+ ]+ },+ {+ "id": 9,+ "name": "Lifecycle Bootstrap",+ "prompt": "Every new session starts inconsistently. Design startup, hook, and handoff lifecycle rules so a new agent can resume safely.",+ "expected_output": "Lifecycle bootstrap design with init.sh, clean-state checks, handoff reads, and hook trust boundaries",+ "files": [],+ "expectations": [+ "init.sh is the standard startup and verification entrypoint",+ "Startup reads AGENTS.md, feature_list.json, progress.md, and handoff when present",+ "End-of-session procedure records evidence, blockers, and next step",+ "Hooks are gated by trust and failure behavior is explicit"+ ]+ },+ {+ "id": 10,+ "name": "Scripted Harness Validation",+ "prompt": "Use the harness-creator scripts to scaffold a harness, validate it, and produce an HTML assessment report for a repository.",+ "expected_output": "Commands using create-harness.mjs, validate-harness.mjs, and render-assessment-html.mjs plus interpretation of score",+ "files": [],+ "expectations": [+ "Uses create-harness.mjs with a target directory",+ "Uses validate-harness.mjs and explains five-subsystem score output",+ "Uses render-assessment-html.mjs or run-benchmark.mjs with --html",+ "Explains that structural benchmark complements but does not replace real before/after agent sessions"+ ]}]}diff --git a/skills/harness-creator/metadata.json b/skills/harness-creator/metadata.jsonindex 8a16b6e..76fcf71 100644--- a/skills/harness-creator/metadata.json+++ b/skills/harness-creator/metadata.json@@ -20,9 +20,11 @@"references": ["references/memory-persistence-pattern.md","references/context-engineering-pattern.md",+ "references/skill-runtime-pattern.md","references/tool-registry-pattern.md","references/multi-agent-pattern.md",- "references/lifecycle-bootstrap-pattern.md"+ "references/lifecycle-bootstrap-pattern.md",+ "references/gotchas.md"],"triggers": ["harness engineering",@@ -42,11 +44,18 @@"permission pipeline"],"bundled": {- "scripts": [],+ "scripts": [+ "scripts/create-harness.mjs",+ "scripts/validate-harness.mjs",+ "scripts/render-assessment-html.mjs",+ "scripts/run-benchmark.mjs",+ "scripts/lib/harness-utils.mjs"+ ],"assets": [],"templates": ["templates/agents.md","templates/feature-list.json",+ "templates/feature-list.schema.json","templates/init.sh","templates/progress.md","templates/session-handoff.md"diff --git a/skills/harness-creator/scripts/create-harness.mjs b/skills/harness-creator/scripts/create-harness.mjsnew file mode 100755index 0000000..1a33a7f--- /dev/null+++ b/skills/harness-creator/scripts/create-harness.mjs@@ -0,0 +1,75 @@+#!/usr/bin/env node+import { chmod, mkdir } from 'node:fs/promises';+import path from 'node:path';+import {+ copyTemplate,+ detectPackageManager,+ detectProject,+ exists,+ initScriptFromCommands,+ parseArgs,+ verificationCommands,+ writeText+} from './lib/harness-utils.mjs';++const args = parseArgs(process.argv.slice(2));++if (args.help) {+ console.log(`Usage: node scripts/create-harness.mjs [--target DIR] [--agent-file AGENTS.md|CLAUDE.md] [--package-manager npm|pnpm|yarn|bun] [--force]++Creates a minimal production harness:+ AGENTS.md or CLAUDE.md+ feature_list.json+ progress.md+ session-handoff.md+ init.sh++Existing files are skipped unless --force is set.`);+ process.exit(0);+}++const target = path.resolve(args.target || args._[0] || process.cwd());+const agentFile = args.agentFile || 'AGENTS.md';+const force = Boolean(args.force);+const project = await detectProject(target);+project.packageManager = detectPackageManager(target, args.packageManager);+const commands = args.commands+ ? String(args.commands).split(',').map((command) => command.trim()).filter(Boolean)+ : verificationCommands(project, args.packageManager);++await mkdir(target, { recursive: true });++const replacements = {+ AGENT_FILE_NAME: agentFile,+ PROJECT_PURPOSE: project.stack === 'generic'+ ? 'Project harness for reliable agent-assisted development.'+ : `Project harness for reliable agent-assisted development in a ${project.stack} codebase.`,+ VERIFICATION_COMMANDS: commands.map((command) => `- \`${command}\``).join('\n'),+ PRIMARY_VERIFICATION_COMMAND: './init.sh'+};++const results = [];+results.push(await copyTemplate('agents.md', path.join(target, agentFile), replacements, { force }));+results.push(await copyTemplate('feature-list.json', path.join(target, 'feature_list.json'), {}, { force }));+results.push(await copyTemplate('progress.md', path.join(target, 'progress.md'), {}, { force }));+results.push(await copyTemplate('session-handoff.md', path.join(target, 'session-handoff.md'), {}, { force }));++const initPath = path.join(target, 'init.sh');+if (force || !await exists(initPath)) {+ await writeText(initPath, initScriptFromCommands(commands));+ await chmod(initPath, 0o755);+ results.push({ path: initPath, status: 'written' });+} else {+ results.push({ path: initPath, status: 'skipped', reason: 'exists' });+}++console.log(`Created harness for ${target}`);+console.log(`Detected stack: ${project.stack}`);+console.log(`Verification commands:`);+for (const command of commands) {+ console.log(` - ${command}`);+}+console.log('');+for (const result of results) {+ console.log(`${result.status.toUpperCase()} ${path.relative(target, result.path)}${result.reason ? ` (${result.reason})` : ''}`);+}diff --git a/skills/harness-creator/scripts/lib/harness-utils.mjs b/skills/harness-creator/scripts/lib/harness-utils.mjsnew file mode 100644index 0000000..fbe3571--- /dev/null+++ b/skills/harness-creator/scripts/lib/harness-utils.mjs@@ -0,0 +1,407 @@+import { existsSync } from 'node:fs';+import { access, chmod, copyFile, mkdir, readFile, readdir, writeFile } from 'node:fs/promises';+import path from 'node:path';+import { fileURLToPath } from 'node:url';++export const SKILL_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..');+export const TEMPLATE_DIR = path.join(SKILL_ROOT, 'templates');+export const SUBSYSTEMS = ['instructions', 'state', 'verification', 'scope', 'lifecycle'];++export function parseArgs(argv) {+ const args = { _: [] };+ for (let i = 0; i < argv.length; i += 1) {+ const token = argv[i];+ if (!token.startsWith('--')) {+ args._.push(token);+ continue;+ }+ const [rawKey, inlineValue] = token.slice(2).split('=', 2);+ const key = rawKey.replace(/-([a-z])/g, (_, letter) => letter.toUpperCase());+ if (inlineValue !== undefined) {+ args[key] = inlineValue;+ } else if (argv[i + 1] && !argv[i + 1].startsWith('--')) {+ args[key] = argv[i + 1];+ i += 1;+ } else {+ args[key] = true;+ }+ }+ return args;+}++export async function exists(filePath) {+ try {+ await access(filePath);+ return true;+ } catch {+ return false;+ }+}++export async function readText(filePath) {+ return readFile(filePath, 'utf8');+}++export async function readJson(filePath) {+ return JSON.parse(await readText(filePath));+}++export async function writeText(filePath, contents) {+ await mkdir(path.dirname(filePath), { recursive: true });+ await writeFile(filePath, contents, 'utf8');+}++export async function copyTemplate(templateName, targetPath, replacements = {}, { force = false } = {}) {+ if (!force && await exists(targetPath)) {+ return { path: targetPath, status: 'skipped', reason: 'exists' };+ }++ let contents = await readText(path.join(TEMPLATE_DIR, templateName));+ for (const [key, value] of Object.entries(replacements)) {+ contents = contents.split(`{{${key}}}`).join(value);+ }+ await writeText(targetPath, contents);+ if (templateName.endsWith('.sh')) {+ await chmod(targetPath, 0o755);+ }+ return { path: targetPath, status: 'written' };+}++export function detectPackageManager(root, explicit) {+ if (explicit) return explicit;+ if (existsSync(path.join(root, 'bun.lockb')) || existsSync(path.join(root, 'bun.lock'))) return 'bun';+ if (existsSync(path.join(root, 'pnpm-lock.yaml'))) return 'pnpm';+ if (existsSync(path.join(root, 'yarn.lock'))) return 'yarn';+ return 'npm';+}++export async function detectProject(root) {+ const files = await listFiles(root, { maxFiles: 800 });+ const has = (name) => files.some((file) => file === name || file.endsWith(`/${name}`));+ const hasPrefix = (prefix) => files.some((file) => file.startsWith(prefix));+ const packageJsonPath = path.join(root, 'package.json');+ const packageJson = await exists(packageJsonPath).then((ok) => ok ? readJson(packageJsonPath) : null);++ let stack = 'generic';+ if (packageJson) {+ const deps = { ...packageJson.dependencies, ...packageJson.devDependencies };+ if (deps.react || hasPrefix('src/renderer')) stack = 'typescript-react';+ else if (deps.typescript || has('tsconfig.json')) stack = 'typescript';+ else stack = 'node';+ } else if (has('pyproject.toml') || has('requirements.txt')) {+ stack = 'python';+ } else if (has('go.mod')) {+ stack = 'go';+ } else if (has('Cargo.toml')) {+ stack = 'rust';+ } else if (has('pom.xml')) {+ stack = 'java-maven';+ } else if (has('build.gradle') || has('build.gradle.kts')) {+ stack = 'java-gradle';+ } else if (files.some((file) => file.endsWith('.csproj') || file.endsWith('.sln'))) {+ stack = 'dotnet';+ }++ return {+ root,+ stack,+ packageJson,+ files,+ packageManager: detectPackageManager(root)+ };+}++export async function listFiles(root, { maxFiles = 1000 } = {}) {+ const ignored = new Set(['.git', 'node_modules', 'dist', 'build', '.next', '.venv', 'venv', '__pycache__']);+ const results = [];++ async function walk(current, relative) {+ if (results.length >= maxFiles) return;+ let entries = [];+ try {+ entries = await readdir(current, { withFileTypes: true });+ } catch {+ return;+ }+ for (const entry of entries) {+ if (results.length >= maxFiles) return;+ if (ignored.has(entry.name)) continue;+ const rel = relative ? `${relative}/${entry.name}` : entry.name;+ const full = path.join(current, entry.name);+ if (entry.isDirectory()) {+ await walk(full, rel);+ } else if (entry.isFile()) {+ results.push(rel);+ }+ }+ }++ await walk(root, '');+ return results.sort();+}++export function verificationCommands(project, explicitPackageManager) {+ const pm = explicitPackageManager || project.packageManager || 'npm';+ const scripts = project.packageJson?.scripts ?? {};+ const run = (script) => {+ if (pm === 'npm') return `npm run ${script}`;+ if (pm === 'yarn') return `yarn ${script}`;+ return `${pm} run ${script}`;+ };++ if (project.stack === 'python') {+ return [+ 'python -m pytest',+ 'python -m compileall .'+ ];+ }++ if (project.stack === 'go') return ['go test ./...'];+ if (project.stack === 'rust') return ['cargo test'];+ if (project.stack === 'java-maven') return ['mvn test'];+ if (project.stack === 'java-gradle') return ['./gradlew test'];+ if (project.stack === 'dotnet') return ['dotnet test'];++ if (!project.packageJson) {+ return [+ 'echo "No package manifest detected; replace this line with your project verification command."'+ ];+ }++ const install = pm === 'npm'+ ? 'npm install'+ : pm === 'yarn'+ ? 'yarn install'+ : `${pm} install`;+ const candidates = [+ scripts.check ? run('check') : null,+ scripts.typecheck ? run('typecheck') : null,+ scripts['type-check'] ? run('type-check') : null,+ scripts.lint ? run('lint') : null,+ scripts.test ? (pm === 'npm' ? 'npm test' : `${pm} test`) : null,+ scripts.build ? run('build') : null+ ].filter(Boolean);++ return [install, ...dedupe(candidates)];+}++export function initScriptFromCommands(commands) {+ const body = commands.map((command) => `echo "=== ${escapeForEcho(command)} ==="\n${command}`).join('\n\n');+ return `#!/bin/bash+set -e++echo "=== Harness Initialization ==="++${body}++echo "=== Verification Complete ==="+echo ""+echo "Next steps:"+echo "1. Read feature_list.json to see current feature state"+echo "2. Pick ONE unfinished feature to work on"+echo "3. Implement only that feature"+echo "4. Re-run verification before claiming done"+`;+}++function escapeForEcho(value) {+ return value.replaceAll('"', '\\"');+}++export function dedupe(values) {+ return [...new Set(values)];+}++export function scoreHarness(files) {+ const byPath = new Map(files.map((file) => [file.path, file.content]));+ const allText = files.map((file) => `${file.path}\n${file.content}`).join('\n\n');+ const agents = byPath.get('AGENTS.md') || byPath.get('CLAUDE.md') || '';+ const featureList = byPath.get('feature_list.json') || byPath.get('feature-list.json') || '';+ const progress = byPath.get('progress.md') || '';+ const init = byPath.get('init.sh') || '';+ const handoff = byPath.get('session-handoff.md') || '';++ const checks = {+ instructions: [+ hasFile(byPath, ['AGENTS.md', 'CLAUDE.md'], 'Agent instruction file exists'),+ textHas(agents, ['Startup Workflow', 'Before writing code'], 'Startup workflow documented'),+ textHas(agents, ['Definition of Done', 'done only when'], 'Definition of done documented'),+ textHas(agents, ['Verification Commands', './init.sh', 'test', 'verify'], 'Verification commands discoverable'),+ textHas(agents, ['feature_list.json', 'progress.md'], 'State artifacts routed from instructions')+ ],+ state: [+ hasFile(byPath, ['feature_list.json', 'feature-list.json'], 'Feature tracker exists'),+ jsonFeatureList(featureList, 'Feature tracker is valid and has feature fields'),+ hasFile(byPath, ['progress.md'], 'Progress log exists'),+ textHas(progress, ['Current State', 'What', 'Next'], 'Progress log supports restart'),+ textHas(handoff || progress, ['Blockers', 'Files', 'Next Session'], 'Handoff captures blockers/files/next step')+ ],+ verification: [+ hasFile(byPath, ['init.sh'], 'Verification entrypoint exists'),+ textHas(init, ['set -e'], 'Verification fails fast'),+ textHas(init + agents, ['test', 'pytest', 'vitest', 'cargo test', 'go test', 'dotnet test'], 'Test command documented'),+ textHas(init + agents, ['build', 'type', 'lint', 'compile'], 'Static/build check documented'),+ textHas(allText, ['Evidence', 'Verification Evidence', 'command and output'], 'Verification evidence is recorded')+ ],+ scope: [+ textHas(agents, ['One feature at a time', 'one-feature-at-a-time'], 'One-feature-at-a-time rule exists'),+ textHas(featureList, ['dependencies'], 'Feature dependencies are tracked'),+ textHas(agents + featureList, ['status'], 'Feature status is explicit'),+ textHas(agents, ['Stay in scope', 'scope'], 'Scope boundary documented'),+ textHas(agents, ['Definition of Done'], 'Completion gate limits scope closure')+ ],+ lifecycle: [+ hasFile(byPath, ['init.sh'], 'Startup script exists'),+ textHas(agents, ['End of Session', 'Before ending'], 'End-of-session procedure exists'),+ hasFile(byPath, ['session-handoff.md'], 'Session handoff template exists'),+ textHas(progress + handoff, ['Last Updated', 'Current Objective', 'Recommended Next Step'], 'Session restart markers exist'),+ textHas(agents + init, ['restartable', 'clean', 'Next steps'], 'Clean restart path documented')+ ]+ };++ const subsystems = Object.fromEntries(Object.entries(checks).map(([name, subsystemChecks]) => {+ const passed = subsystemChecks.filter((check) => check.pass).length;+ const score = Math.max(1, Math.round((passed / subsystemChecks.length) * 5));+ return [name, {+ score,+ passed,+ total: subsystemChecks.length,+ checks: subsystemChecks+ }];+ }));++ const total = Object.values(subsystems).reduce((sum, item) => sum + item.score, 0);+ const overall = Math.round((total / (SUBSYSTEMS.length * 5)) * 100);+ const bottleneck = Object.entries(subsystems).sort((a, b) => a[1].score - b[1].score)[0][0];+ return { overall, bottleneck, subsystems };+}++function hasFile(byPath, names, message) {+ return { pass: names.some((name) => byPath.has(name)), message };+}++function textHas(text, needles, message) {+ const lower = text.toLowerCase();+ return { pass: needles.some((needle) => lower.includes(needle.toLowerCase())), message };+}++function jsonFeatureList(text, message) {+ try {+ const parsed = JSON.parse(text);+ const valid = Array.isArray(parsed.features) && parsed.features.every((feature) =>+ typeof feature.id === 'string'+ && typeof feature.name === 'string'+ && typeof feature.description === 'string'+ && typeof feature.status === 'string'+ );+ return { pass: valid, message };+ } catch {+ return { pass: false, message };+ }+}++export async function loadHarnessFiles(root) {+ const candidates = [+ 'AGENTS.md',+ 'CLAUDE.md',+ 'feature_list.json',+ 'feature-list.json',+ 'progress.md',+ 'session-handoff.md',+ 'init.sh'+ ];+ const files = [];+ for (const candidate of candidates) {+ const fullPath = path.join(root, candidate);+ if (await exists(fullPath)) {+ files.push({ path: candidate, content: await readText(fullPath) });+ }+ }+ return files;+}++export function formatScoreReport(result, root = '.') {+ const lines = [+ `Harness validation for ${root}`,+ `Overall: ${result.overall}/100`,+ `Bottleneck: ${result.bottleneck}`,+ ''+ ];++ for (const [name, subsystem] of Object.entries(result.subsystems)) {+ lines.push(`${name}: ${subsystem.score}/5 (${subsystem.passed}/${subsystem.total})`);+ for (const check of subsystem.checks) {+ lines.push(` ${check.pass ? 'PASS' : 'FAIL'} ${check.message}`);+ }+ lines.push('');+ }+ return lines.join('\n');+}++export function htmlReport(result, title = 'Harness Assessment') {+ const rows = Object.entries(result.subsystems).map(([name, subsystem]) => {+ const checks = subsystem.checks.map((check) =>+ `<li class="${check.pass ? 'pass' : 'fail'}">${check.pass ? 'PASS' : 'FAIL'} ${escapeHtml(check.message)}</li>`+ ).join('');+ return `<section>+ <h2>${escapeHtml(name)} <span>${subsystem.score}/5</span></h2>+ <ul>${checks}</ul>+ </section>`;+ }).join('\n');++ return `<!doctype html>+<html lang="en">+<head>+ <meta charset="utf-8">+ <meta name="viewport" content="width=device-width, initial-scale=1">+ <title>${escapeHtml(title)}</title>+ <style>+ body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; margin: 32px; color: #172026; background: #f7f8fa; }+ main { max-width: 960px; margin: 0 auto; }+ header { margin-bottom: 24px; }+ h1 { margin: 0 0 8px; font-size: 32px; }+ .summary { display: flex; gap: 16px; flex-wrap: wrap; margin: 20px 0; }+ .metric { background: white; border: 1px solid #d9dee5; border-radius: 8px; padding: 16px 18px; min-width: 180px; }+ .metric strong { display: block; font-size: 28px; margin-top: 4px; }+ section { background: white; border: 1px solid #d9dee5; border-radius: 8px; margin: 14px 0; padding: 16px 18px; }+ h2 { margin: 0 0 10px; font-size: 20px; display: flex; justify-content: space-between; }+ ul { margin: 0; padding-left: 20px; }+ li { margin: 6px 0; }+ .pass { color: #126c43; }+ .fail { color: #a23020; }+ </style>+</head>+<body>+ <main>+ <header>+ <h1>${escapeHtml(title)}</h1>+ <p>Five-subsystem harness validation report.</p>+ <div class="summary">+ <div class="metric">Overall<strong>${result.overall}/100</strong></div>+ <div class="metric">Bottleneck<strong>${escapeHtml(result.bottleneck)}</strong></div>+ </div>+ </header>+ ${rows}+ </main>+</body>+</html>+`;+}++function escapeHtml(value) {+ return String(value)+ .replaceAll('&', '&')+ .replaceAll('<', '<')+ .replaceAll('>', '>')+ .replaceAll('"', '"')+ .replaceAll("'", ''');+}++export async function copyFileSafe(source, target, { force = false } = {}) {+ if (!force && await exists(target)) {+ return { path: target, status: 'skipped', reason: 'exists' };+ }+ await mkdir(path.dirname(target), { recursive: true });+ await copyFile(source, target);+ return { path: target, status: 'written' };+}diff --git a/skills/harness-creator/scripts/render-assessment-html.mjs b/skills/harness-creator/scripts/render-assessment-html.mjsnew file mode 100755index 0000000..4d8d85f--- /dev/null+++ b/skills/harness-creator/scripts/render-assessment-html.mjs@@ -0,0 +1,26 @@+#!/usr/bin/env node+import path from 'node:path';+import {+ htmlReport,+ loadHarnessFiles,+ parseArgs,+ scoreHarness,+ writeText+} from './lib/harness-utils.mjs';++const args = parseArgs(process.argv.slice(2));++if (args.help) {+ console.log(`Usage: node scripts/render-assessment-html.mjs [--target DIR] [--output FILE]++Renders the five-subsystem harness assessment as a standalone HTML file.`);+ process.exit(0);+}++const target = path.resolve(args.target || args._[0] || process.cwd());+const output = path.resolve(args.output || path.join(target, 'harness-assessment.html'));+const result = scoreHarness(await loadHarnessFiles(target));++await writeText(output, htmlReport(result, `Harness Assessment: ${path.basename(target)}`));+console.log(`HTML report written to ${output}`);+console.log(`Overall: ${result.overall}/100`);diff --git a/skills/harness-creator/scripts/run-benchmark.mjs b/skills/harness-creator/scripts/run-benchmark.mjsnew file mode 100755index 0000000..2e6ab0f--- /dev/null+++ b/skills/harness-creator/scripts/run-benchmark.mjs@@ -0,0 +1,121 @@+#!/usr/bin/env node+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import {+ formatScoreReport,+ htmlReport,+ loadHarnessFiles,+ parseArgs,+ readJson,+ scoreHarness,+ writeText+} from './lib/harness-utils.mjs';++const args = parseArgs(process.argv.slice(2));+const scriptDir = path.dirname(fileURLToPath(import.meta.url));+const skillRoot = path.resolve(scriptDir, '..');++if (args.help) {+ console.log(`Usage: node scripts/run-benchmark.mjs [--target DIR] [--output FILE] [--html FILE]++Runs a lightweight harness benchmark:+ 1. Scores the current target harness.+ 2. Checks eval coverage in evals/evals.json.+ 3. Produces a JSON report and optional HTML report.++This is a structural benchmark, not an LLM judge. Use it before/after real agent sessions.`);+ process.exit(0);+}++const target = path.resolve(args.target || args._[0] || process.cwd());+const output = path.resolve(args.output || path.join(target, 'harness-benchmark.json'));+const evalPath = path.resolve(args.evals || path.join(skillRoot, 'evals', 'evals.json'));++const harnessResult = scoreHarness(await loadHarnessFiles(target));+const evals = await readJson(evalPath);+const evalResult = scoreEvals(evals);+const report = {+ generatedAt: new Date().toISOString(),+ target,+ harness: harnessResult,+ evals: evalResult,+ recommendation: recommend(harnessResult, evalResult)+};++await writeText(output, `${JSON.stringify(report, null, 2)}\n`);+console.log(`Benchmark report written to ${output}`);+console.log('');+console.log(formatScoreReport(harnessResult, target));+console.log(`Eval coverage: ${evalResult.score}/100 (${evalResult.passed}/${evalResult.total})`);+console.log(`Recommendation: ${report.recommendation}`);++if (args.html) {+ const htmlPath = path.resolve(args.html);+ await writeText(htmlPath, renderBenchmarkHtml(report));+ console.log(`HTML benchmark report written to ${htmlPath}`);+}++if (harnessResult.overall < Number(args.minScore || 70) || evalResult.score < Number(args.minEvalScore || 80)) {+ process.exitCode = 1;+}++function scoreEvals(evalsJson) {+ const cases = Array.isArray(evalsJson.evals) ? evalsJson.evals : [];+ const checks = [];+ checks.push({ pass: cases.length >= 10, message: 'At least 10 eval cases' });+ checks.push({ pass: cases.some((item) => /minimal|creation/i.test(item.name)), message: 'Covers minimal harness creation' });+ checks.push({ pass: cases.some((item) => /session|continuity/i.test(item.name)), message: 'Covers session continuity' });+ checks.push({ pass: cases.some((item) => /assessment|score/i.test(item.name)), message: 'Covers harness assessment' });+ checks.push({ pass: cases.some((item) => /verification/i.test(item.name)), message: 'Covers verification workflow' });+ checks.push({ pass: cases.some((item) => /memory/i.test(item.name)), message: 'Covers memory taxonomy' });+ checks.push({ pass: cases.some((item) => /tool|permission|safety/i.test(item.name)), message: 'Covers tool safety' });+ checks.push({ pass: cases.some((item) => /multi-agent|delegation|coordination/i.test(item.name)), message: 'Covers multi-agent coordination' });+ checks.push({ pass: cases.every((item) => item.prompt && item.expected_output && Array.isArray(item.expectations)), message: 'Each eval has prompt, expected output, expectations' });+ checks.push({ pass: cases.every((item) => item.expectations?.length >= 3), message: 'Each eval has at least three expectation checks' });++ const passed = checks.filter((check) => check.pass).length;+ return {+ score: Math.round((passed / checks.length) * 100),+ passed,+ total: checks.length,+ cases: cases.length,+ checks+ };+}++function recommend(harnessResult, evalResult) {+ if (harnessResult.overall >= 85 && evalResult.score >= 90) {+ return 'Ready for realistic before/after agent-session benchmarking.';+ }+ if (harnessResult.overall < 70) {+ return `Improve the ${harnessResult.bottleneck} subsystem before benchmarking agent behavior.`;+ }+ if (evalResult.score < 80) {+ return 'Expand eval coverage before treating benchmark results as representative.';+ }+ return 'Usable, with some gaps worth tightening after first real sessions.';+}++function renderBenchmarkHtml(report) {+ const evalHtml = htmlReport(report.harness, `Harness Benchmark: ${path.basename(report.target)}`)+ .replace('</main>', `<section>+ <h2>Eval Coverage <span>${report.evals.score}/100</span></h2>+ <p>${report.evals.passed}/${report.evals.total} benchmark checks passed across ${report.evals.cases} eval cases.</p>+ <ul>${report.evals.checks.map((check) => `<li class="${check.pass ? 'pass' : 'fail'}">${check.pass ? 'PASS' : 'FAIL'} ${escapeHtml(check.message)}</li>`).join('')}</ul>+ </section>+ <section>+ <h2>Recommendation</h2>+ <p>${escapeHtml(report.recommendation)}</p>+ </section>+ </main>`);+ return evalHtml;+}++function escapeHtml(value) {+ return String(value)+ .replaceAll('&', '&')+ .replaceAll('<', '<')+ .replaceAll('>', '>')+ .replaceAll('"', '"')+ .replaceAll("'", ''');+}diff --git a/skills/harness-creator/scripts/validate-harness.mjs b/skills/harness-creator/scripts/validate-harness.mjsnew file mode 100755index 0000000..3ae16d3--- /dev/null+++ b/skills/harness-creator/scripts/validate-harness.mjs@@ -0,0 +1,43 @@+#!/usr/bin/env node+import path from 'node:path';+import {+ formatScoreReport,+ htmlReport,+ loadHarnessFiles,+ parseArgs,+ scoreHarness,+ writeText+} from './lib/harness-utils.mjs';++const args = parseArgs(process.argv.slice(2));++if (args.help) {+ console.log(`Usage: node scripts/validate-harness.mjs [--target DIR] [--json] [--html FILE]++Scores a project harness across five subsystems:+ instructions, state, verification, scope, lifecycle++Exit code is 0 when the harness scores at least --min-score (default 70).`);+ process.exit(0);+}++const target = path.resolve(args.target || args._[0] || process.cwd());+const minScore = Number(args.minScore || 70);+const files = await loadHarnessFiles(target);+const result = scoreHarness(files);++if (args.html) {+ const htmlPath = path.resolve(args.html);+ await writeText(htmlPath, htmlReport(result, `Harness Assessment: ${path.basename(target)}`));+ console.log(`HTML report written to ${htmlPath}`);+}++if (args.json) {+ console.log(JSON.stringify(result, null, 2));+} else {+ console.log(formatScoreReport(result, target));+}++if (result.overall < minScore) {+ process.exitCode = 1;+}diff --git a/skills/harness-creator/templates/agents.md b/skills/harness-creator/templates/agents.mdindex bae9a82..0b41e94 100644--- a/skills/harness-creator/templates/agents.md+++ b/skills/harness-creator/templates/agents.md@@ -1,6 +1,6 @@-# AGENTS.md+# {{AGENT_FILE_NAME}}-[One-sentence project purpose]+{{PROJECT_PURPOSE}}## Startup Workflow@@ -8,7 +8,7 @@ Before writing code:1. **Confirm working directory** with `pwd`2. **Read this file** completely-3. **Read [ARCHITECTURE.md / CLAUDE.md]** for system map and hard rules+3. **Read project docs if present** (`docs/ARCHITECTURE.md`, `docs/PRODUCT.md`, README, or equivalent)4. **Run `./init.sh`** to verify environment is healthy5. **Read `feature_list.json`** to see current feature state6. **Review recent commits** with `git log --oneline -5`@@ -53,16 +53,16 @@ Before ending a session:```bash# Full verification (recommended)-./init.sh--# Individual checks-npm install && npm run check && npm test+{{PRIMARY_VERIFICATION_COMMAND}}```+Required checks:+{{VERIFICATION_COMMANDS}}+## EscalationIf you encounter:-- **Architecture decisions**: Consult [ARCHITECTURE.md] or ask user-- **Unclear requirements**: Check [docs/PRODUCT.md] or ask user+- **Architecture decisions**: Consult project architecture docs if present, otherwise ask user+- **Unclear requirements**: Check product/requirements docs if present, otherwise ask user- **Repeated test failures**: Update progress, flag for human review- **Scope ambiguity**: Re-read `feature_list.json` for definition of donediff --git a/skills/harness-creator/templates/feature-list.json b/skills/harness-creator/templates/feature-list.jsonindex d0f369f..fff05b0 100644--- a/skills/harness-creator/templates/feature-list.json+++ b/skills/harness-creator/templates/feature-list.json@@ -2,40 +2,40 @@"features": [{"id": "feat-001",- "name": "Document Import",- "description": "Allow users to import PDF and TXT documents from local filesystem",+ "name": "Project Setup",+ "description": "Confirm the project can install dependencies, run verification, and start from a clean checkout","dependencies": [],"status": "not-started","evidence": ""},{"id": "feat-002",- "name": "Document Chunking",- "description": "Split imported documents into ~500 character chunks with metadata (word count, line count, paragraph count)",+ "name": "First User-Facing Feature",+ "description": "Replace this placeholder with the first concrete behavior the agent should implement","dependencies": ["feat-001"],"status": "not-started","evidence": ""},{"id": "feat-003",- "name": "Indexing Status UI",- "description": "Display indexing progress in status bar: total docs, indexed docs, last sync time",+ "name": "Verification Coverage",+ "description": "Add or confirm tests, type checks, linting, or manual verification for the active feature","dependencies": ["feat-002"],"status": "not-started","evidence": ""},{"id": "feat-004",- "name": "Grounded Q&A",- "description": "Run AI-powered Q&A over indexed content with citations to source documents",- "dependencies": ["feat-002"],+ "name": "Documentation Update",+ "description": "Update README, architecture notes, or product docs affected by the implemented feature",+ "dependencies": ["feat-003"],"status": "not-started","evidence": ""},{"id": "feat-005",- "name": "Conversation History",- "description": "Display Q&A conversation in chat-style interface with copy/export functionality",+ "name": "Cleanup and Handoff",+ "description": "Record verification evidence, update progress.md, and leave a clear next-session path","dependencies": ["feat-004"],"status": "not-started","evidence": ""diff --git a/skills/harness-creator/templates/init.sh b/skills/harness-creator/templates/init.shindex e866eae..a305a37 100644--- a/skills/harness-creator/templates/init.sh+++ b/skills/harness-creator/templates/init.sh@@ -3,10 +3,68 @@ set -eecho "=== Harness Initialization ==="-npm install-npm run check-npm test-npm run build+if [ -f package.json ]; then+ if [ -f pnpm-lock.yaml ]; then+ PM="pnpm"+ elif [ -f yarn.lock ]; then+ PM="yarn"+ elif [ -f bun.lock ] || [ -f bun.lockb ]; then+ PM="bun"+ else+ PM="npm"+ fi++ echo "=== Installing dependencies with $PM ==="+ if [ "$PM" = "npm" ]; then+ npm install+ else+ "$PM" install+ fi++ node -e "const s=require('./package.json').scripts||{}; process.exit(s.check||s.typecheck||s['type-check']?0:1)" && {+ if node -e "const s=require('./package.json').scripts||{}; process.exit(s.check?0:1)"; then+ [ "$PM" = "npm" ] && npm run check || "$PM" run check+ elif node -e "const s=require('./package.json').scripts||{}; process.exit(s.typecheck?0:1)"; then+ [ "$PM" = "npm" ] && npm run typecheck || "$PM" run typecheck+ else+ [ "$PM" = "npm" ] && npm run type-check || "$PM" run type-check+ fi+ }++ node -e "const s=require('./package.json').scripts||{}; process.exit(s.lint?0:1)" && {+ [ "$PM" = "npm" ] && npm run lint || "$PM" run lint+ }++ node -e "const s=require('./package.json').scripts||{}; process.exit(s.test?0:1)" && {+ [ "$PM" = "npm" ] && npm test || "$PM" test+ }++ node -e "const s=require('./package.json').scripts||{}; process.exit(s.build?0:1)" && {+ [ "$PM" = "npm" ] && npm run build || "$PM" run build+ }+elif [ -f pyproject.toml ] || [ -f requirements.txt ]; then+ echo "=== Running Python verification ==="+ python -m pytest+ python -m compileall .+elif [ -f go.mod ]; then+ echo "=== Running Go verification ==="+ go test ./...+elif [ -f Cargo.toml ]; then+ echo "=== Running Rust verification ==="+ cargo test+elif [ -f pom.xml ]; then+ echo "=== Running Maven verification ==="+ mvn test+elif [ -f build.gradle ] || [ -f build.gradle.kts ]; then+ echo "=== Running Gradle verification ==="+ ./gradlew test+elif ls *.csproj *.sln >/dev/null 2>&1; then+ echo "=== Running .NET verification ==="+ dotnet test+else+ echo "No recognized package manifest detected."+ echo "Replace this section with the project's verification commands."+fiecho "=== Verification Complete ==="echo ""