Skip to content

Awesome list for AI agent harness engineering: tools, patterns, evals, memory, MCP, permissions, observability, and orchestration.

[Evals] Add LevelBench-100 multi-file CLI and browser harness benchmark

Sam Young1dcf2f2e4682
diff --git a/README.md b/README.md
index 6640f3c..8988370 100644
--- a/README.md
+++ b/README.md
@@ -610,6 +610,7 @@ Real repositories worth studying — each with a note on *why* it's worth your t
- [Live-SWE-agent: First Live Software Agent with Self-Evolving Scaffold](https://arxiv.org/html/2511.13646v3) — Demonstrates end-to-end harness design for production software engineering: 77.4% solve rate on SWE-bench Verified (vs. 50% from human contractors). The key insight is the self-evolving scaffold — the harness itself adapts based on failure signals rather than requiring manual engineering each time a new task class emerges. Shows the upper-bound of what specialized harness design can achieve.
- [OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models](https://arxiv.org/abs/2604.10866) — April 2026 benchmark that evaluates agents on 12 real-world professional occupations (software engineer, data scientist, financial analyst, etc.) using language world models to simulate realistic work environments. Agents are scored on task completion, efficiency, and professional standards adherence. The important harness contribution is the environment-as-evaluator pattern: instead of static test cases, the benchmark uses dynamic language-based world models that respond to agent actions with realistic consequences — a reference design for evaluating harnesses in open-ended, profession-specific domains.
- [STATE-Bench](https://github.com/microsoft/STATE-Bench) — Microsoft's open-source benchmark (May 2026) measuring whether agents actually improve with experience on 450 realistic enterprise tasks across customer support, travel, and shopping. The first eval to treat memory as an independent variable with explicit learning tracks, providing a concrete way to compare memory architectures beyond simple recall metrics. ![Stars](https://img.shields.io/github/stars/microsoft/STATE-Bench?style=flat-square&label=★&color=yellow)
+- [LevelBench-100](https://github.com/levelcfo/levelbench-100) — Empirical benchmark isolating base model capability vs. CLI harness architecture (Claude Code, Codex CLI, Gemini CLI, Antigravity CLI) and deterministic verification playbooks across 100 multi-file accounting close checks, 15 SMB close workbooks, and 14 browser-automation portal gates. Demonstrates that while raw frontier models top out at 71.0% with a 28.6% bad-entry rate, multi-file CLI harnesses reach 84.0% unassisted and 100.0% (0.0% bad entries) when paired with deterministic verification checklists and human-in-the-loop hold gates. ![Stars](https://img.shields.io/github/stars/levelcfo/levelbench-100?style=flat-square&label=★&color=yellow)
---