Skip to content

Awesome list for AI agent harness engineering: tools, patterns, evals, memory, MCP, permissions, observability, and orchestration.

Add Causal Improvement Graph (arXiv 2610.05039) to Generators & Meta-Harnesses

ai-boost7eec5e7ef8e0
diff --git a/README.md b/README.md
index a9178fe..bea4e1c 100644
--- a/README.md
+++ b/README.md
@@ -464,6 +464,7 @@ Real repositories worth studying — each with a note on *why* it's worth your t
- [MemoHarness: Agent Harnesses That Learn from Experience](https://arxiv.org/abs/2607.14159) — July 2026 framework that decomposes the harness into six control dimensions along the inference flow (context assembly, tool interaction, generation control, orchestration, memory management, output processing) and adapts the configuration per case by retrieving from a dual-layer experience bank of per-case failure diagnoses and distilled global patterns — no labels, feedback, or search rounds at test time. Beats Codex on Terminal-Bench (0.806 vs 0.722) at lower cost ($6.89 vs $10.28, with ~94% of input tokens cacheable), transfers +0.098 on average across six base models, and ships an [open-source implementation](https://github.com/HowieHwong/MemoHarness). The clearest demonstration that a harness can be a retrieval-adapted artifact rather than a fixed configuration or an expensive optimization loop — and the six-dimension decomposition is a reusable taxonomy for auditing any harness's editable surface.
- [Harness-Zero: Harness Distillation via Agent-as-Harness](https://arxiv.org/abs/2609.24974) — September 2026 method (Peking University, Google, HKUST) for making harness gains portable: a harnessing agent guided by an evolved reference harness reviews and corrects the student's *own* trajectories in the target harness's action space before execution, turning harness guidance into SFT demonstrations. The critical findings — the distilled model under a minimal harness beats the base model with the optimized harness attached (44.3% vs 41.7% macro-average across SpreadsheetBench, AppWorld, and USPTO), and correcting the student's own trajectories transfers far better than training on stronger models' trajectories — define the boundary conditions for when harness intelligence belongs in weights versus in scaffolding. [Open-source implementation](https://github.com/metaevo-ai/harness-zero). ![Stars](https://img.shields.io/github/stars/metaevo-ai/harness-zero?style=flat-square&label=★&color=yellow)
- [SBCO: Self-Supervised, Verifier-Grounded Harness Optimization for Planning Agents](https://arxiv.org/abs/2608.10157) — August 2026 (Samsung Research America) method for domains where self-referential harness improvement breaks down because task competence doesn't imply self-modification competence (unlike coding). SBCO jointly learns a decomposed bank of per-constraint verifiers and a harness policy via block coordinate ascent — a fixed meta-agent grades its own outputs in a closed loop with no human labels — matching or beating the customized Huxley Gödel Machine baseline on travel planning and shopping while using 4–5.5× less compute. The 86%-of-gains-from-high-F1-verifiers analysis is a concrete diagnostic for which constraints repay verifier investment, and the cross-model policy transfer (GPT-5.4-mini → Mistral) shows the learned verifier bank is a portable harness asset rather than model-specific tuning.
+- [Causal Improvement Graph for Agentic Harness Optimization](https://arxiv.org/abs/2610.05039) — October 2026 meta-harness framework that externalizes the improvement state of harness optimization into a persistent graph: Evidence, Hypothesis, Intervention, and Outcome nodes linked so each proposer builds on the *relations* among prior findings instead of re-deriving them from an ever-growing raw history. Directly targets the scaling failure of proposer-centric meta-harnesses (Meta-Harness, Self-Harness), where the proposer's context fills with candidate noise and improvement stalls — reporting stronger discovered harnesses than prior meta-harness baselines with robustness to the choice of task solver and proposer. The graph-of-improvement-state idea is a concrete answer to how harness search can keep accumulating structure across hundreds of iterations rather than starting each round from a blank context.
- [HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?](https://arxiv.org/abs/2609.01437) — September 2026 benchmark that treats the harness itself as the artifact under evaluation: given a minimal seed, creator LLMs must build a complete execution system from scratch, then evolve it from downstream feedback, scored on capability and execution-token cost. Across four domains and five benchmarks (2,207 instances), generated harnesses fall substantially behind human-engineered references on code and deep-research tasks but match or beat them on writing and ML experimentation — and evolution gains are unstable, only partially transfer to unseen tasks, and depend heavily on the executing model. The necessary counterweight to this section's positive results: it quantifies exactly where autonomous harness creation is not yet competitive, and shows that evolution ≠ generalization.
### Demo Harnesses