Skip to content

Awesome list for AI agent harness engineering: tools, patterns, evals, memory, MCP, permissions, observability, and orchestration.

Add Microsoft STATE-Bench: open-source benchmark for agent memory

ai-boostbdeee114c2f8
diff --git a/README.md b/README.md
index fd94d5d..563c40d 100644
--- a/README.md
+++ b/README.md
@@ -463,6 +463,7 @@ Real repositories worth studying — each with a note on *why* it's worth your t
- [Amazon Bedrock AgentCore Evaluations Is Now Generally Available](https://aws.amazon.com/about-aws/whats-new/2026/03/agentcore-evaluations-generally-available/) — AWS's March 31, 2026 GA launch matters because it operationalizes agent evals as an infrastructure service: trajectory scoring, task completion checks, and model-graded assessments are wired into the same platform that hosts agents. Worth adding because it shows how evaluation stops being an offline benchmark and becomes part of the runtime control plane teams can standardize on.
- [Live-SWE-agent: First Live Software Agent with Self-Evolving Scaffold](https://arxiv.org/html/2511.13646v3) — Demonstrates end-to-end harness design for production software engineering: 77.4% solve rate on SWE-bench Verified (vs. 50% from human contractors). The key insight is the self-evolving scaffold — the harness itself adapts based on failure signals rather than requiring manual engineering each time a new task class emerges. Shows the upper-bound of what specialized harness design can achieve.
- [OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models](https://arxiv.org/abs/2604.10866) — April 2026 benchmark that evaluates agents on 12 real-world professional occupations (software engineer, data scientist, financial analyst, etc.) using language world models to simulate realistic work environments. Agents are scored on task completion, efficiency, and professional standards adherence. The important harness contribution is the environment-as-evaluator pattern: instead of static test cases, the benchmark uses dynamic language-based world models that respond to agent actions with realistic consequences — a reference design for evaluating harnesses in open-ended, profession-specific domains.
+- [STATE-Bench](https://github.com/microsoft/STATE-Bench) — Microsoft's open-source benchmark (May 2026) measuring whether agents actually improve with experience on 450 realistic enterprise tasks across customer support, travel, and shopping. The first eval to treat memory as an independent variable with explicit learning tracks, providing a concrete way to compare memory architectures beyond simple recall metrics. ![Stars](https://img.shields.io/github/stars/microsoft/STATE-Bench?style=flat-square&label=★&color=yellow)
---