Awesome list for AI agent harness engineering: tools, patterns, evals, memory, MCP, permissions, observability, and orchestration.
Python agent-harnessagent-memoryagent-orchestrationai-agent-harnessai-agentsawesome-listcontext-engineeringharness-engineeringmcp
Post-merge fixes + daily addition: add star badges to AEGIS (#363) and Speakeasy gram (#365) entries, correct MCP Security Testing Checklist note to eight steps (#367); add AutoCompact (arXiv 2610.02163) to Context Delivery & Compaction — October 2026 work turning compaction timing from a fixed token threshold into a learned agent behavior (judge-corrected trajectories + SFT + joint RL; +9.2% on SWE-bench Verified)
ai-boost6d4db43f59ed
diff --git a/README.md b/README.mdindex c6d7e19..e9f378c 100644--- a/README.md+++ b/README.md@@ -182,6 +182,7 @@ Harness components organized by the problem they solve, not by vendor.- [Boris Cherny: Building Claude Code (YC Startup School, July 2026)](https://www.ycombinator.com/library/UN-boris-cherny-building-claude-code) — The Claude Code creator's account of shipping Opus 5: Anthropic deleted ~80% of the system prompt because it encoded corrections for behaviors the model no longer has, with no eval regression. The ablation methodology — strip the prompt entirely, add back only lines that fix repeated failures, repeat at every model release — is the sharpest published answer to harness rot: accumulated instructions are bets against model capability, and they go stale.- [Context management in agent harnesses: memory, files, and subagents](https://arize.com/blog/context-management-in-agent-harnesses/) — Arize's April 2026 comparative teardown of how Pi, OpenClaw, Claude Code, and Letta each manage context as an active resource: hard read caps with pagination, LLM compaction triggered at token thresholds with tool-call boundary safety, oversized-result persistence to disk, and subagent context isolation. The striking finding is convergence — four independent harnesses plus Arize's own production agent arrived at the same mechanisms — making it the clearest evidence-based catalog of which context-management patterns are becoming standard rather than experimental.- [Context Engineering in 2026: Why We Stopped Compacting Our Agent's Context](https://www.louisbouchard.ai/context-engineering-2026/) — August 2026 controlled study on a production AI tutor (660 turns, planted-fact probes) that inverts the compaction default: keeping full history reached 100% memory recall at $0.11/turn while the compacting setup managed 58% recall at $0.24/turn, because summarization rewrites the cached prefix and destroys cache hits. The decision framework — name the real constraint first (window size → retrieval, bill → model choice, quality rot → targeted fixes) and prefer cache-preserving truncation and tool-output caps over rewriting history — is the clearest evidence-based argument that compaction should be conditional, not automatic.+- [AutoCompact: Learning When to Compact Context in Long-Horizon Coding Agents](https://arxiv.org/abs/2610.02163) — October 2026 paper that turns compaction timing from a fixed harness policy into a learned agent behavior: a judge corrects compaction timing, summary quality, and post-compaction actions in collected trajectories, then supervised fine-tuning plus joint RL optimizes coding and compaction together. +9.2% absolute on SWE-bench Verified and +5.0% on SWE-PolyBench Verified — including in a 256K window where overflow never occurs, proving the model learns *when* forgetting is safe rather than merely reacting to pressure. The key harness insight: token thresholds are a stopgap; the right compaction trigger is semantic (is the work at a safe boundary?), and that judgment is trainable like any other agent skill.### Tool Design@@ -366,7 +367,7 @@ Harness components organized by the problem they solve, not by vendor.### Observability & Tracing-- [AEGIS](https://github.com/antropos17/Aegis) — Windows-first desktop monitor for local agent processes, configured file activity, TCP endpoints and audit exports. Useful for inspecting a harness's host-side activity; monitoring is observational by default, while selected-action policies and MCP gateways need explicit setup.+- [AEGIS](https://github.com/antropos17/Aegis) — Windows-first desktop monitor for local agent processes, configured file activity, TCP endpoints and audit exports. Useful for inspecting a harness's host-side activity; monitoring is observational by default, while selected-action policies and MCP gateways need explicit setup. - [OpenLLMetry](https://github.com/traceloop/openllmetry) — OpenTelemetry-based instrumentation for LLM calls and agent steps: adds trace spans to every inference and tool call without modifying business logic. The cleanest way to bring the existing OTEL ecosystem (Grafana, Datadog, Jaeger) to a harness. - [Arize Phoenix](https://github.com/Arize-ai/phoenix) — Self-hostable trace UI and eval runtime for agent workflows. Lets harness engineers audit and replay every reasoning step and tool call offline, without sending data to a third-party cloud. - [Opik](https://github.com/comet-ml/opik) — Comet's open-source AI observability and evaluation platform: deep tracing of LLM calls, conversation logging, and agent activity, plus built-in eval metrics, prompt versioning, guardrails, and the Opik Agent Optimizer. Worth including because it unifies observability, verification, and optimization in one self-hostable stack rather than stitching together separate tools. @@ -584,9 +585,9 @@ Real repositories worth studying — each with a note on *why* it's worth your t- [Trustworthy agents in practice](https://www.anthropic.com/research/trustworthy-agents) — Anthropic's April 2026 framework for governing autonomous agents through five principles: human control, value alignment, secure interactions, transparency, and privacy. The most complete published treatment of how to design harness-level governance that keeps pace with increasing agent capability and autonomy.- [RAMPART](https://github.com/microsoft/RAMPART) — Pytest-native safety and security testing framework for agentic AI that turns red-team findings into repeatable CI tests. Supports statistical trials (e.g., "safe in 95% of runs") rather than single-shot pass/fail, making it the first concrete tool for treating agent safety as an engineering discipline rather than a post-hoc audit. - [aiming-lab/AutoHarness](https://github.com/aiming-lab/AutoHarness) — Open-source Python governance harness that wraps any OpenAI-compatible client with a configurable tool-approval pipeline, prompt-injection defense, secret-exposure checks, and JSONL audit trails. Demonstrates how to productize "Agent = Model + Harness" into a drop-in governance layer rather than scattering guardrails across application code. -- [Speakeasy control plane](https://github.com/speakeasy-api/gram) — MCP gateway that scopes agent tool access by identity, team, and role, with security policy enforced on each action and an audit trail. Useful as a reference for putting authorization at the tool boundary instead of relying on model prompts to restrict access.+- [Speakeasy control plane](https://github.com/speakeasy-api/gram) — MCP gateway that scopes agent tool access by identity, team, and role, with security policy enforced on each action and an audit trail. Useful as a reference for putting authorization at the tool boundary instead of relying on model prompts to restrict access. -- [MCP Security Testing Checklist](https://loki-intelligence.com/blog/mcp-security-testing-checklist) — Twelve-step practitioner verification checklist for MCP server and tool-harness boundaries: OAuth/token scoping, schema and description poisoning, dynamic discovery freezes, SSRF/path-traversal argument validation, cross-tenant isolation, and secret redaction in tool responses. Provides a concrete pre-release audit rubric for verifying that MCP integrations cannot be abused to cross harness trust boundaries.+- [MCP Security Testing Checklist](https://loki-intelligence.com/blog/mcp-security-testing-checklist) — Eight-step practitioner verification checklist for MCP server and tool-harness boundaries: OAuth/token scoping, schema and description poisoning, dynamic discovery freezes, SSRF/path-traversal argument validation, cross-tenant isolation, and secret redaction in tool responses. Provides a concrete pre-release audit rubric for verifying that MCP integrations cannot be abused to cross harness trust boundaries.---