Skip to content

Awesome list for AI agent harness engineering: tools, patterns, evals, memory, MCP, permissions, observability, and orchestration.

Round 7: add 3 latest harness engineering resources

ai-boost275b8fa4eade
diff --git a/README.md b/README.md
index dbf3df1..11c436b 100644
--- a/README.md
+++ b/README.md
@@ -248,6 +248,7 @@ Harness components organized by the problem they solve, not by vendor.
### Debugging & Developer Experience
- [AgentOps](https://github.com/AgentOps-AI/agentops) — Open-source agent engineering platform (YC W24) with session replay, cost tracking, and failure detection across 10+ frameworks including CrewAI, LangGraph, and OpenAI Agents SDK. The step-by-step execution graph and cross-session metrics make it the most practical debugging layer for multi-agent systems in production. ![Stars](https://img.shields.io/github/stars/AgentOps-AI/agentops?style=flat-square&label=★&color=yellow)
+- [Syncause/debug-skill](https://github.com/Syncause/debug-skill) — April 2026 agent debugging skill that stops guesswork with runtime evidence. Uses background tracing (Runtime Facts) to capture the exact execution path leading to failures, then constrains the agent to cite specific data points (stack traces, variable snapshots) before proposing fixes. Moves agent debugging from "patch and pray" to evidence-based repair with reviewable results. ![Stars](https://img.shields.io/github/stars/Syncause/debug-skill?style=flat-square&label=★&color=yellow)
- [AgentRx: Systematic Debugging for AI Agents](https://www.microsoft.com/en-us/research/blog/systematic-debugging-for-ai-agents-introducing-the-agentrx-framework/) — Framework for automated root-cause analysis of agent failures: trajectory normalization, constraint synthesis from tool schemas, and constraint-guided evaluation. Achieves 23.6% better failure localization than existing approaches with a 115-trajectory annotated benchmark. Shifts agent debugging from manual log inspection to systematic constraint-based diagnosis — a reference design for harness-level observability that surfaces *why* an agent failed, not just *that* it failed. ![Stars](https://img.shields.io/github/stars/microsoft/AgentRx?style=flat-square&label=★&color=yellow)
- [Debugging Deep Agents with LangSmith](https://blog.langchain.com/debugging-deep-agents-with-langsmith/) — Addresses the core problem of debugging agents that run for minutes, span hundreds of steps, and produce massive traces no human can manually scan. Introduces Polly (an AI assistant that analyzes traces to surface root causes) and langsmith-fetch (CLI for piping trace data to coding agents). Key insight: debugging deep agents requires AI-assisted trace analysis — the volume of data these systems produce exceeds human capacity.
- [Where LLM Agents Fail and How They Can Learn From Failures (AgentDebug)](https://arxiv.org/abs/2509.25370) — ICLR 2026 paper introducing the Agent Error Taxonomy — a modular classification covering memory, reflection, planning, action, and system-level failures. The AgentDebug framework isolates root-cause failures and provides corrective feedback, achieving +24% higher all-correct accuracy. The Agent Error Benchmark (annotated trajectories from ALFWorld, GAIA, WebShop) is the first systematic failure dataset for agent debugging.
@@ -257,6 +258,7 @@ Harness components organized by the problem they solve, not by vendor.
### Human-in-the-Loop
+- [aws-samples/sample-human-in-the-loop-patterns](https://github.com/aws-samples/sample-human-in-the-loop-patterns) — March 2026 AWS reference implementation demonstrating four distinct HITL patterns for sensitive agent tool calls: Hook System (centralized blanket policy), Tool Context (per-tool fine-grained), Step Functions (async third-party approval via SNS), and MCP Elicitation (protocol-native real-time interactive approval). The most concrete production guide for choosing the right HITL architecture based on approval latency, trust boundaries, and integration constraints. ![Stars](https://img.shields.io/github/stars/aws-samples/sample-human-in-the-loop-patterns?style=flat-square&label=★&color=yellow)
- [HITL Protocol](https://github.com/rotorstar/hitl-protocol) — Open standard (v0.8, February 2026) for human decisions in agent workflows: HTTP 202 + review URL pattern connecting services, agents, and humans across any messaging channel. No SDK required — ~15 lines of code for agents, reference implementations in Express/Hono/Next.js/FastAPI for services, and 13 end-to-end flows including escalation and hybrid approval.
- [LangGraph — Human-in-the-Loop Concepts](https://langchain-ai.github.io/langgraph/concepts/human_in_the_loop/) — Systematic treatment of interrupt, breakpoint, and approve patterns: how to pause an agent mid-loop, persist state, and resume after human review. Directly addresses the harness engineering challenge of inserting human gates into long-running workflows.
- [AutoGen — Human-in-the-Loop](https://microsoft.github.io/autogen/0.2/docs/tutorial/human-in-the-loop/) — Explains `human_input_mode` (NEVER / TERMINATE / ALWAYS) and the UserProxyAgent as an approval gate. The most concrete implementation reference for adding human review nodes to a multi-agent conversation harness.
@@ -287,6 +289,7 @@ Real repositories worth studying — each with a note on *why* it's worth your t
- [everything-claude-code](https://github.com/affaan-m/everything-claude-code) — Anthropic Hackathon Winner (140K+ stars). The agent harness performance optimization system: skills, instincts, memory optimization, continuous learning, security scanning, and research-first development. Production-ready agents, skills, hooks, rules, and MCP configurations evolved over 10+ months of intensive daily use building real products. Works across Claude Code, Codex, Cursor, OpenCode, and Gemini. ![Stars](https://img.shields.io/github/stars/affaan-m/everything-claude-code?style=flat-square&label=★&color=yellow)
- [Claude Agent SDK](https://platform.claude.com/docs/en/agent-sdk/overview) — Anthropic's official SDK that exposes Claude Code's entire harness as a programmable API: built-in tool execution loop, `PreToolUse`/`PostToolUse` hooks for interception, subagent definitions, `allowedTools` permission control, and session resumption. The highest-leverage starting point for building a production harness — you inherit the entire tool execution layer rather than implementing it. ![Stars](https://img.shields.io/github/stars/anthropics/claude-agent-sdk-python?style=flat-square&label=★&color=yellow)
- [revfactory/harness](https://github.com/revfactory/harness) — A meta-skill that generates domain-specific agent teams and the skills they use. Good example of harness-as-code, where the harness itself is produced by an agent. ![Stars](https://img.shields.io/github/stars/revfactory/harness?style=flat-square&label=★&color=yellow)
+- [raphaelchristi/harness-evolver](https://github.com/raphaelchristi/harness-evolver) — March 2026 Claude Code plugin that autonomously evolves LLM agent harnesses using multi-agent proposers in isolated git worktrees, LangSmith-backed evaluation, and regression guards. Iterates on prompts, routing, retrieval, and orchestration code based on full-trace counterfactual diagnosis. The most practical published implementation of the Meta-Harness outer-loop optimization paradigm. ![Stars](https://img.shields.io/github/stars/raphaelchristi/harness-evolver?style=flat-square&label=★&color=yellow)
- [Meta-Harness: End-to-End Optimization of Model Harnesses](https://arxiv.org/abs/2603.28052) — Treats the entire harness (system prompt, tool definitions, context management, completion logic) as a joint optimization target rather than hand-tuning each piece. The key insight: give the proposer agent filesystem access to all prior harness candidates, scores, and execution traces — 10M-token diagnostic context vs. the 26K in prior work — so it can trace failures back to specific harness decisions.
- [HyperAgents: Self-Improving AI Systems](https://pooya.blog/blog/hyperagents-self-improving-ai-meta-research-2026/) — Meta's framework integrating task-solving and meta-level improvement into a unified, editable program with metacognitive self-modification. Improved paper-review tasks from 0.0 to 0.710, transferred to Olympiad math grading at 0.630 improvement@50 score. Shows how agents can be designed to modify their own harness (prompts, tools, strategy) based on execution history — the ultimate meta-harness where the agent itself evolves the scaffolding.
- [AutoAgent](https://github.com/kevinrgu/autoagent) — Open-source library (April 2026) that automates the harness engineering loop itself: give it a task and a benchmark, and it iterates overnight on system prompts, tool configurations, agent orchestration, and routing — keeping or discarding each change based on score. In a 24-hour run, hit #1 on SpreadsheetBench (96.5%) and the top GPT-5 score on TerminalBench (55.1%), beating every hand-engineered entry. The `program.md` separation of concerns (human writes the directive, agent engineers the harness) is the most practical meta-harness pattern published so far. ![Stars](https://img.shields.io/github/stars/kevinrgu/autoagent?style=flat-square&label=★&color=yellow)