Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
Update docs: cross-agent memory, plugin learn architecture, dedup
chopratejasdd488fdaaf89
diff --git a/README.md b/README.mdindex 19807d4f8..8ee4a38c1 100644--- a/README.md+++ b/README.md@@ -130,15 +130,16 @@ Works with any language, any tool, any framework. **[Proxy docs](docs/proxy.md)*### Coding agents — one command```bash-headroom wrap claude # Starts proxy + launches Claude Code-headroom wrap codex # Starts proxy + launches OpenAI Codex CLI-headroom wrap aider # Starts proxy + launches Aider-headroom wrap cursor # Starts proxy + prints Cursor config-headroom wrap openclaw # Installs + configures OpenClaw plugin-headroom unwrap openclaw # Disables plugin + restores legacy engine+headroom wrap claude # Starts proxy + launches Claude Code+headroom wrap codex # Starts proxy + launches OpenAI Codex CLI+headroom wrap aider # Starts proxy + launches Aider+headroom wrap cursor # Starts proxy + prints Cursor config+headroom wrap openclaw # Installs + configures OpenClaw plugin+headroom wrap claude --memory # With persistent cross-agent memory+headroom wrap codex --memory # Shares the same memory store```-Headroom starts a proxy, points your tool at it, and compresses everything automatically.+Headroom starts a proxy, points your tool at it, and compresses everything automatically. Add `--memory` for persistent memory that's shared across agents.### Multi-agent — SharedContext@@ -270,15 +271,24 @@ Auto-detects what's in your context — JSON arrays, code, logs, plain text —Stabilizes message prefixes so your provider's KV cache actually works. Claude offers a 90% read discount on cached prefixes — but almost no framework takes advantage of it. Headroom does.+### Cross-Agent Memory++```bash+headroom wrap claude --memory # Claude with persistent memory+headroom wrap codex --memory # Codex shares the SAME memory store+```++Claude saves a fact, Codex reads it back. All agents sharing one proxy share one memory — project-scoped, user-isolated, with agent provenance tracking and automatic deduplication. No SDK changes needed. **[Memory docs](docs/memory.md)**+### Failure Learning```bash-headroom learn # Analyze past Claude Code sessions, show recommendations-headroom learn --apply # Write learnings to CLAUDE.md and MEMORY.md-headroom learn --all --apply # Learn across all your projects+headroom learn # Auto-detect agent (Claude, Codex, Gemini)+headroom learn --apply # Write learnings to agent-native files+headroom learn --agent codex --all # Analyze all Codex sessions```-Reads your conversation history, finds every failed tool call, correlates it with what eventually succeeded, and writes specific corrections into your project files. Next session starts smarter. **[Learn docs](docs/learn.md)**+Plugin-based: reads conversation history from Claude Code, Codex, or Gemini CLI. Finds failure patterns, correlates with successes, writes corrections to CLAUDE.md / AGENTS.md / GEMINI.md. External plugins via entry points. **[Learn docs](docs/learn.md)**<p align="center"><img src="headroom_learn.gif" alt="headroom learn demo" width="800">@@ -302,10 +312,10 @@ Reads your conversation history, finds every failed tool call, correlates it wit| **CacheAligner** | Stabilizes prefixes for provider KV cache hits || **IntelligentContext** | Score-based context management with learned importance || **Image Compression** | 40-90% token reduction via trained ML router |-| **Memory** | Persistent memory across conversations |+| **Memory** | Cross-agent persistent memory — Claude saves, Codex reads it back. Agent provenance + auto-dedup || **Compression Hooks** | Customize compression with pre/post hooks || **Read Lifecycle** | Detects stale/superseded Read outputs, replaces with CCR markers |-| **`headroom learn`** | Analyzes past failures, writes project-specific learnings to CLAUDE.md/MEMORY.md |+| **`headroom learn`** | Plugin-based failure learning for Claude Code, Codex, Gemini CLI (extensible via entry points) || **`headroom wrap`** | One-command setup for Claude Code, Codex, Aider, Cursor || **SharedContext** | Compressed inter-agent context sharing for multi-agent workflows || **MCP Tools** | headroom_compress, headroom_retrieve, headroom_stats for Claude Code/Cursor |@@ -480,11 +490,11 @@ Python 3.10+| [Latency Benchmarks](docs/LATENCY_BENCHMARKS.md) | Compression overhead & cost-benefit analysis || [Limitations](docs/LIMITATIONS.md) | When compression helps, when it doesn't || [Evals Framework](headroom/evals/README.md) | Prove compression preserves accuracy |-| [Memory](docs/memory.md) | Persistent memory |+| [Memory](docs/memory.md) | Cross-agent persistent memory with provenance + dedup || [Agno](docs/agno.md) | Agno agent framework || [MCP](docs/mcp.md) | Context engineering toolkit (compress, retrieve, stats) || [SharedContext](docs/shared-context.md) | Compressed inter-agent context sharing |-| [Learn](docs/learn.md) | Offline failure learning for coding agents |+| [Learn](docs/learn.md) | Plugin-based failure learning (Claude, Codex, Gemini, extensible) || [Configuration](docs/configuration.md) | All options |---diff --git a/docs/learn.md b/docs/learn.mdindex dce072a81..ce5b1b028 100644--- a/docs/learn.md+++ b/docs/learn.md@@ -21,16 +21,18 @@ headroom learn --all --apply## How It Works```-Past Sessions → Scanner → Analyzer → Writer → CLAUDE.md / MEMORY.md+Past Sessions → Plugin → Analyzer → Writer → Agent-native context file│ │ ││ │ └─ Writes marker-delimited sections│ │ (replaced on re-run, not duplicated)│ │- │ └─ Success Correlation: for each failure,- │ finds what succeeded and extracts the diff+ │ └─ LLM-based analysis: finds failure patterns,+ │ success correlations, and actionable rules│- └─ Reads ~/.claude/projects/*.jsonl- (extensible to Cursor, Codex, etc.)+ └─ Plugin reads agent-specific logs:+ • Claude Code: ~/.claude/projects/*.jsonl+ • Codex: ~/.codex/sessions/*.json+ • Gemini CLI: ~/.gemini/tmp/*/chats/session-*.json```### Success Correlation@@ -95,12 +97,12 @@ Commands repeatedly rejected — model should suggest them to the user instead.## Where Learnings Go-| Pattern | Destination | Why |-|---------|-------------|-----|-| Environment, paths, search scope, commands, large files | **CLAUDE.md** | Stable project facts, version-controllable |-| Missing paths, retry patterns, permissions | **MEMORY.md** | May change, agent-specific |+| Pattern | Claude Code | Codex | Gemini CLI |+|---------|-------------|-------|-----------|+| Environment, paths, commands | **CLAUDE.md** | **AGENTS.md** | **GEMINI.md** |+| Retry patterns, permissions | **MEMORY.md** | **instructions.md** | **GEMINI.md** |-CLAUDE.md lives in your project directory. MEMORY.md lives in `~/.claude/projects/*/memory/`.+Output files are agent-native: Claude Code uses CLAUDE.md/MEMORY.md, Codex uses AGENTS.md, Gemini uses GEMINI.md. The same learnings, written to the format each agent reads.## Marker-Based Updates@@ -116,25 +118,30 @@ Headroom manages a clearly-delimited section in each file:On re-run, only the content between markers is replaced. Your existing file content is preserved.-## Architecture+## Architecture (Plugin System)++Headroom Learn uses a plugin architecture where each agent is a self-contained plugin:```-Scanner (adapter) → Analyzer (generic) → Writer (adapter)-├── ClaudeCodeScanner ├── EnvironmentAnalyzer ├── ClaudeCodeWriter-├── (CursorScanner) ├── StructureAnalyzer ├── (CursorWriter)-└── (GenericScanner) ├── CommandAnalyzer └── (GenericWriter)- ├── RetryAnalyzer- └── CrossSessionAnalyzer+Plugin Registry (auto-discovered)+├── ClaudeCodePlugin → Analyzer (LLM) → ClaudeCodeWriter → CLAUDE.md / MEMORY.md+├── CodexPlugin → Analyzer (LLM) → CodexWriter → AGENTS.md / instructions.md+├── GeminiPlugin → Analyzer (LLM) → GeminiWriter → GEMINI.md+└── (your plugin) → Analyzer (LLM) → (your writer) → (your file)```-**Scanners** read tool-specific log formats and produce normalized `ToolCall` sequences.-**Analyzers** work on `ToolCall` — same analysis for any agent system.-**Writers** output to tool-specific context injection mechanisms.+**Plugins** bundle scanning, detection, and writing for one agent. Built-in plugins are auto-discovered from `headroom.learn.plugins.*`. External plugins register via the `headroom.learn_plugin` entry point.++**The Analyzer** is shared — it uses an LLM (Sonnet, GPT-4o, or Gemini Flash) to find patterns. Same analysis for any agent.++### Adding Support for a New Agent-To add support for a new agent (e.g., Cursor):-1. Write `CursorScanner(ConversationScanner)` — reads Cursor's log format-2. Write `CursorWriter(ContextWriter)` — writes to `.cursorrules`-3. Same analyzers, same models, same recommendations+1. Create `headroom/learn/plugins/myagent.py`+2. Implement `LearnPlugin` + `ConversationScanner` (scanner + writer + detection)+3. Add `plugin = MyAgentPlugin()` at module scope+4. Done — `headroom learn --agent myagent` works automatically++Or install an external plugin: `pip install headroom-learn-cursor` (registers via entry point).## CLI Reference@@ -142,12 +149,22 @@ To add support for a new agent (e.g., Cursor):headroom learn [OPTIONS]Options:- --project PATH Project directory to analyze (default: current directory)- --all Analyze all discovered projects- --apply Write recommendations (default: dry-run)- --claude-dir PATH Path to .claude directory (default: ~/.claude)+ --project PATH Project directory (default: current directory)+ --all Analyze all discovered projects+ --apply Write recommendations (default: dry-run)+ --agent [auto|claude|codex|gemini]+ Which agent to analyze (default: auto-detect)+ --model TEXT LLM for analysis (default: auto from API keys)```+### Supported Agents++| Agent | Scanner | Writer | Output Files |+|-------|---------|--------|-------------|+| **Claude Code** | Reads `~/.claude/projects/*.jsonl` | ClaudeCodeWriter | CLAUDE.md, MEMORY.md |+| **OpenAI Codex** | Reads `~/.codex/sessions/*.json` | CodexWriter | AGENTS.md, instructions.md |+| **Gemini CLI** | Reads `~/.gemini/tmp/*/chats/session-*.json` | GeminiWriter | GEMINI.md |+## Real-World ResultsTested on 67,583 tool calls across 23 projects:diff --git a/docs/memory.md b/docs/memory.mdindex 717cb0b84..ce440a726 100644--- a/docs/memory.md+++ b/docs/memory.md@@ -18,6 +18,10 @@ This is *temporal compression* - instead of carrying 10,000 tokens of conversati| Feature | Headroom | Letta (MemGPT) | Mem0 ||---------|----------|----------------|------|+| **Cross-Agent Memory** | Any agent shares one DB via proxy | Per-agent only | Per-user, no cross-agent |+| **Agent Provenance** | Tracks which agent saved/updated each memory | No | No |+| **LLM-Mediated Dedup** | Piggybacks on user's own LLM for merge decisions | No | Separate LLM call ($) |+| **Transparent Proxy** | Zero code changes — just route through proxy | Requires agent framework | Requires SDK integration || **Hierarchical Scoping** | User → Session → Agent → Turn | Flat (per-agent) | Flat (per-user) || **Temporal Versioning** | Full supersession chains | No | No || **Zero-Latency Extraction** | Inline (Letta-style) | Inline | Separate call |@@ -29,6 +33,112 @@ This is *temporal compression* - instead of carrying 10,000 tokens of conversati---+## Cross-Agent Memory (Proxy)++The most powerful way to use memory: **any agent that routes through the proxy shares the same memory store.** Claude saves a fact, Codex reads it back. Zero configuration needed.++```bash+# Start the proxy with memory enabled+headroom proxy --memory++# Or use wrap (auto-starts proxy)+headroom wrap claude --memory # Claude Code with persistent memory+headroom wrap codex --memory # Codex with the SAME memory store+headroom wrap aider --memory # Aider shares it too+```++### How It Works++```+Claude Code Codex CLI Gemini CLI+ │ │ │+ └── /v1/messages ──┐ └── /v1/chat/completions ──┤ └── /generateContent ──┐+ │ │ │+ ▼ ▼ ▼+ ┌──────────────────────────────────────────────────────────────────┐+ │ Headroom Proxy (--memory) │+ │ │+ │ 1. Search memory DB for relevant context │+ │ 2. Inject memories as system context (provider-native format) │+ │ 3. Add memory_save/search/update/delete tools │+ │ 4. Forward to upstream LLM │+ │ 5. Handle memory tool calls in response │+ │ 6. Async background dedup (>92% cosine → auto-remove) │+ │ │+ └──────────────────────┬───────────────────────────────────────────┘+ │+ ▼+ .headroom/memory.db+ (project-scoped SQLite)+```++### Project-Scoped Database++Memory is stored per-project at `{cwd}/.headroom/memory.db`. Each project has its own memory — no cross-project contamination. Override with `--memory-db-path` for a custom location.++### User Identity++User ID is auto-detected from `$USER` (your OS username). Override per-request with the `x-headroom-user-id` header. All memories are scoped to the user — multiple developers on the same project have separate memory stores.++### Agent Provenance++Every memory tracks which agent created or updated it:++```json+{+ "content": "Project uses alembic for migrations",+ "metadata": {+ "source_agent": "claude",+ "source_provider": "anthropic",+ "created_via": "tool_call",+ "created_at_utc": "2026-04-10T17:30:00Z"+ }+}+```++When an agent updates a memory, the update is tracked:++```json+{+ "reason": "Updated by codex via openai: Added version info"+}+```++### Intelligent Deduplication++When the LLM calls `memory_save`, headroom:++1. **Saves immediately** (zero latency)+2. **Searches for similar existing memories** (cosine similarity)+3. **Returns an enriched hint** if duplicates found:++```json+{+ "status": "saved",+ "memory_id": "abc123",+ "note": "Similar memory exists (id: def456, 89% match, saved by codex):+ 'DB migration tool is alembic'. Call memory_update('def456',+ '<merged content>') to consolidate."+}+```++The LLM then decides whether to merge — using the user's own LLM, not a separate model. No extra cost to headroom.++4. **Background auto-dedup**: If similarity >92%, the older duplicate is automatically removed (async, non-blocking).++### Supported Providers++Memory works with ALL providers routing through the proxy:++| Provider | Context Injection | Memory Tools | Format |+|----------|-------------------|--------------|--------|+| **Anthropic** (Claude) | System parameter | Anthropic tool_use | Native |+| **OpenAI** (Codex, GPT) | System message | OpenAI function calling | Native |+| **Gemini** | systemInstruction | functionDeclarations | Native |+| **Any OpenAI-compatible** | System message | Function calling | OpenAI format |++---+## Quick Start```python@@ -443,6 +553,10 @@ Headroom Memory uses **Protocol interfaces** (ports) for all components, enablin| Feature | Headroom | Letta | Mem0 ||---------|:--------:|:-----:|:----:|+| Cross-agent sharing (proxy) | ✅ | ❌ | ❌ |+| Agent provenance tracking | ✅ | ❌ | ❌ |+| LLM-mediated dedup (no extra cost) | ✅ | ❌ | ❌ (uses separate LLM) |+| Transparent proxy (zero code) | ✅ | ❌ | ❌ || Hierarchical scoping | ✅ | ❌ | ❌ || Temporal versioning | ✅ | ❌ | ❌ || Zero-latency extraction | ✅ | ✅ | ❌ |