Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
Update token headroom mode spec with review fixes
chopratejasf9476a1c46ce
diff --git a/docs/superpowers/specs/2026-03-19-token-headroom-mode-design.md b/docs/superpowers/specs/2026-03-19-token-headroom-mode-design.mdindex d5b1d7185..940c1bf52 100644--- a/docs/superpowers/specs/2026-03-19-token-headroom-mode-design.md+++ b/docs/superpowers/specs/2026-03-19-token-headroom-mode-design.md@@ -38,16 +38,28 @@ class CompressionCache:"""cache: dict[str, str] # content_hash → compressed_contenttoken_savings: dict[str, int] # content_hash → tokens saved- max_entries: int = 2000 # LRU eviction beyond this+ max_entries: int = 10000 # LRU eviction beyond thisdef get_compressed(self, content_hash: str) -> str | Nonedef store_compressed(self, content_hash: str, compressed: str, tokens_saved: int)def compute_frozen_count(self, messages: list[dict]) -> int+ def update_from_result(self, originals: list[dict], compressed: list[dict]) -> None```**Cache key**: SHA-256 of original content. Same file content across turns = same hash = cache hit. Edited file = different hash = cache miss = fresh compression.-**Frozen count calculation**: Count consecutive messages from the start that have cache hits. First cache miss = end of stable compressed prefix. This handles Claude Code dropping messages (breaks the consecutive run, stops the frozen prefix there).+**Max entries**: 10,000 (not 2,000). Long coding sessions can generate thousands of tool results. Evicting cache entries forces re-compression, defeating the re-freeze optimization. 10K entries at ~1KB average compressed text = ~10MB — acceptable.++**Frozen count calculation**: Count consecutive messages from the start that are "stable" (either have cache hits OR are non-compressible messages like user/assistant/tool-use). The key rule:++- **Tool-result messages**: Stable if content hash is in cache (compressed version available).+- **User, assistant, system, tool-use messages**: Always considered stable (they pass through unchanged, so they won't bust the prefix cache).++First *unstable* message (tool-result with cache miss) = end of frozen prefix. This prevents the frozen count from being 0 just because message 0 is a user message.++This handles Claude Code dropping messages: if a cached tool-result is dropped, the next message may not be in cache, breaking the consecutive run and stopping the frozen prefix there.++**`update_from_result` contract**: Performs index-aligned comparison between `originals` and `compressed`. For each index where content differs, stores the mapping original_hash → compressed_content. Assumes the pipeline does NOT reorder or merge messages (this is a pipeline contract that holds today and must be preserved).### Pipeline Flow in Token Headroom Mode@@ -70,6 +82,27 @@ Zone 3: Protection window → Pass through unchanged (last 30% of messages)**Critical invariant**: The proxy ONLY operates on messages Claude Code actually sends. It never adds, re-inserts, or reorders messages. If Claude Code drops a message via its own context management, the proxy does not re-add it.+### Zone 3 (Protection Window) Definition++Zone 3 protects the **last 30% of excluded-tool messages** (Read, Glob, Grep, Write, Edit results), not 30% of all messages. This matches the existing `protect_recent_reads_fraction` semantics in ContentRouter. User messages, assistant messages, and tool-use blocks are always protected regardless of position — they don't factor into the 30% calculation.++Example: 100 messages, 40 of which are Read/Glob/Grep results → last 12 excluded-tool results protected (30% of 40). All user/assistant messages protected independently.++### Config Plumbing++`protect_recent_reads_fraction` is a `ContentRouterConfig` instance attribute, not a per-call kwarg. It is set **at proxy startup** based on mode:++```python+# In HeadroomProxy.__init__ or _build_pipelines:+if self.config.mode == "token_headroom":+ router_config.protect_recent_reads_fraction = 0.3+ router_config.ccr_ttl = 14400+else:+ router_config.protect_recent_reads_fraction = 0.0 # protect all (current default)+```++Similarly, `force_compress_threshold` on `PrefixFreezeConfig` is set to `0.0` in token_headroom mode (always allow compression through the cache). This avoids two competing mechanisms for cache-bust decisions.+### Integration PointIn `server.py`, `_handle_anthropic_request` (line ~2071):@@ -81,17 +114,16 @@ if self.config.mode == "token_headroom":# Zone 1: Swap cached compressed versionsworking_messages = comp_cache.apply_cached(messages)- # Calculate re-freeze boundary (consecutive cache hits from start)+ # Calculate re-freeze boundary (consecutive stable messages from start)frozen_count = comp_cache.compute_frozen_count(messages)- # Zone 2+3: Pipeline compresses Zone 2, skips Zone 3+ # Zone 2+3: Pipeline compresses Zone 2, skips Zone 3 (protected)result = self.anthropic_pipeline.apply(working_messages, model,frozen_message_count=frozen_count,- protect_recent_fraction=0.3,)- # Cache newly compressed messages+ # Cache newly compressed messages (index-aligned diff)comp_cache.update_from_result(messages, result.messages)else:# Current cost_savings behavior — unchanged@@ -168,7 +200,18 @@ Headroom proxy v0.4.6 | Mode: token_headroomCompression cache: active```-## Also Fix: Missing Model Entry+## Observability++In token_headroom mode, the `/stats` endpoint and startup logs must expose:++- **Per-turn zone breakdown**: Zone 1/2/3 message counts and token counts+- **Compression cache stats**: entries, hits, misses, hit rate+- **Net token savings**: tokens saved by compression minus tokens lost to cache bust write premium+- **CCR store stats**: entries, memory usage, eviction count++The CCR store uses TTL-based eviction (entries expire after 4 hours). The `CompressionCache` uses LRU eviction at 10K entries. Both report eviction counts in stats so operators can detect if bounds are too tight.++## Also Fix: Missing Model Entry (Separate Task)`claude-opus-4-6` is not in `ANTHROPIC_CONTEXT_LIMITS`. Falls back to pattern match → 200K. Real context is 1M. Add it with correct limit. (Separate from token_headroom mode but discovered during investigation.)@@ -186,7 +229,11 @@ Headroom proxy v0.4.6 | Mode: token_headroom6. **Compression makes content larger**: ContentRouter already checks compression_ratio >= min_ratio and passes through unchanged. Cache only stores actual reductions.-7. **Claude Code drops messages**: Cache entries go unused (LRU eviction eventually). Proxy never re-inserts dropped messages. Frozen count stops at first cache miss, preventing stale frozen prefix.+7. **Claude Code drops messages**: Cache entries go unused (LRU eviction eventually). Proxy never re-inserts dropped messages. Frozen count stops at first unstable message, preventing stale frozen prefix.++8. **Mode switch mid-session**: If proxy restarts with a different mode, the CompressionCache is empty (new process) and the prefix tracker is reset. In token_headroom mode, the first turn does a full compression pass (edge case 5). In cost_savings mode, the prefix tracker rebuilds from the first API response. No stale state carries over.++9. **Large Zone 2 on first turn after --resume**: Many tool results age out simultaneously. ContentRouter processes them sequentially. If compression latency exceeds 10 seconds, log a warning but do not skip — the user explicitly chose token_headroom mode and the one-time cost is justified. Subsequent turns are fast (cache hits).## Testing Strategy@@ -221,9 +268,8 @@ Headroom proxy v0.4.6 | Mode: token_headroom## Files to Modify1. **New file**: `headroom/cache/compression_cache.py` — CompressionCache class-2. **Modify**: `headroom/proxy/server.py` — ProxyConfig.mode, pipeline flow branch, session store-3. **Modify**: `headroom/transforms/pipeline.py` — Accept protect_recent_fraction kwarg-4. **Modify**: `headroom/transforms/content_router.py` — Use protect_recent_fraction from kwargs-5. **Modify**: `headroom/providers/anthropic.py` — Add claude-opus-4-6 to ANTHROPIC_CONTEXT_LIMITS-6. **New file**: `tests/test_compression_cache.py` — Unit tests-7. **New file**: `tests/test_token_headroom_mode.py` — Integration + E2E tests+2. **Modify**: `headroom/proxy/server.py` — ProxyConfig.mode, mode-based config at startup, pipeline flow branch, CompressionCache session store, stats endpoint additions+3. **Modify**: `headroom/proxy/server.py` — Set `ContentRouterConfig.protect_recent_reads_fraction` and `PrefixFreezeConfig.force_compress_threshold` at startup based on mode+4. **Modify**: `headroom/providers/anthropic.py` — Add claude-opus-4-6 to ANTHROPIC_CONTEXT_LIMITS (separate commit)+5. **New file**: `tests/test_compression_cache.py` — Unit tests for CompressionCache+6. **New file**: `tests/test_token_headroom_mode.py` — Integration + E2E tests