Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
docs: drop effort-routing text that #3377 removed (#4449)
danieldutu3a38d0b93b22
diff --git a/README.md b/README.mdindex adacc2f21..dd44d0dfc 100644--- a/README.md+++ b/README.md@@ -183,13 +183,10 @@ back at you, and deep reasoning spent on routine steps like reading a file.Headroom trims it from the proxy, with no change to your code:- **Verbosity steering** appends a short "be terse, don't restate context" note to the *end* of the system prompt, so your prompt cache still hits.-- **Effort routing** dials thinking effort down when a turn is only the model resuming after a tool result — a file read, a passing test. New questions and errors keep full effort.-Both apply to Anthropic `/v1/messages` and to OpenAI-compatible-`/v1/chat/completions` and `/v1/responses`. Effort routing uses-`reasoning_effort` on OpenAI and `thinking.budget_tokens` / `output_config.effort`-on Anthropic, with the same clamp-only invariant and the same `output_shaper:*`-labels on both paths.+It applies to Anthropic `/v1/messages` and to OpenAI-compatible+`/v1/chat/completions` and `/v1/responses`, with the same `output_shaper:*`+labels on all paths.```bashexport HEADROOM_OUTPUT_SHAPER=1 # off by defaultdiff --git a/headroom/agent_savings.py b/headroom/agent_savings.pyindex 000464c1a..3d5211bd5 100644--- a/headroom/agent_savings.py+++ b/headroom/agent_savings.py@@ -173,7 +173,7 @@ _PROFILES: dict[str, AgentSavingsProfile] = {# Coding posture: defer non-core tool schemas, dedupe re-reads, extend# lossy coverage when lossless found nothing, and NEVER lossy-compress a# file read (the agent patches exact bytes). CCR stays ON (unset) so any- # lossy loss is recoverable. Effort router off; low block floor so modest+ # lossy loss is recoverable. Low block floor so modest# deltas are eligible.tool_search=True,cross_turn_dedup=True,diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.pyindex c30010b33..26937b867 100644--- a/headroom/proxy/handlers/anthropic.py+++ b/headroom/proxy/handlers/anthropic.py@@ -121,7 +121,7 @@ def _thinking_tokens_for(payload: object) -> ThinkingTokens:Anthropic reports no thinking count in ``usage``, so the count is derivedfrom the response's own thinking blocks and marked inferred. Without the- split, reasoning-effort routing and verbosity steering move the same+ split, thinking-budget changes and verbosity steering move the samecounter and neither can be attributed.Never raises: token accounting is bookkeeping, and a failure here must@@ -3648,8 +3648,7 @@ class AnthropicHandlerMixin:logger.debug("consistency token re-count skipped", exc_info=True)# Output shaping (opt-in via HEADROOM_OUTPUT_SHAPER): verbosity- # steering appended to the system-prompt tail + effort routing on- # mechanical tool_result continuations. Runs after every other+ # steering appended to the system-prompt tail. Runs after every other# body mutation so the turn classifier sees the final messages,# and respects the same bypass header as compression.if not _bypass:diff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.pyindex 741326e46..00499e80c 100644--- a/headroom/proxy/handlers/openai.py+++ b/headroom/proxy/handlers/openai.py@@ -861,7 +861,7 @@ def _shape_openai_responses_payload(The Responses counterpart of the Anthropic handler's shaping block:conversation-stable holdout assignment, stratum labelling for theoutput-savings ledger, then verbosity steering on the ``instructions``- tail and ``reasoning.effort`` routing on mechanical continuations.+ tail.Returns ``(labels, mutated)``. ``labels`` are carried on the transformschannel for both arms (the control arm's stratum label is what feeds thediff --git a/headroom/proxy/outcome.py b/headroom/proxy/outcome.pyindex 837544683..036c6cc0d 100644--- a/headroom/proxy/outcome.py+++ b/headroom/proxy/outcome.py@@ -124,7 +124,7 @@ class RequestOutcome:# ── Output composition ────────────────────────────────────────────# ``output_tokens`` pools two quantities that different levers move in- # opposite directions: reasoning-effort routing cuts thinking, verbosity+ # opposite directions: thinking-budget levers cut thinking, verbosity# steering cuts visible text. Summed, neither can be attributed — a turn# whose thinking dropped 4,000 tokens while its prose grew 200 is# indistinguishable from one where nothing happened.@@ -250,7 +250,7 @@ class RequestOutcome:The quantity verbosity steering actually targets. Callers must handle``None`` rather than defaulting it to ``output_tokens``: on a providerthat reports no thinking count, treating the whole output as visible- would credit steering with reductions that reasoning-effort routing+ would credit steering with reductions that a lower thinking budgetproduced.Clamped at zero because an inferred count comes from Headroom'sdiff --git a/headroom/proxy/output_shaper.py b/headroom/proxy/output_shaper.pyindex 7987f7c0e..4dabb5187 100644--- a/headroom/proxy/output_shaper.py+++ b/headroom/proxy/output_shaper.py@@ -9,18 +9,7 @@ output tokens, so every lever here works by reshaping the request:provider prefix cache is preserved). Five levels, from "no ceremony" tofull caveman.-2. Effort routing — agentic loops are mostly mechanical continuations (the- last message is a clean tool_result: a file read, a passing test). Thinking- bills as output tokens, and harnesses like Claude Code pin- ``output_config.effort`` at ``xhigh`` for every turn. On turns classified- as mechanical we lower an explicitly-present effort; on errors or new user- asks we leave it alone. For legacy models still sending- ``thinking.budget_tokens`` we clamp the budget to the API floor instead.-Safety rules (each prevents a concrete failure mode):-- Never INJECT ``output_config.effort`` where the client didn't send it —- models without effort support 400 on it. Lowering an existing value is- always valid.- Never toggle ``thinking.type`` — disabling thinking while history carriesthinking blocks 400s on some models, and the toggle busts the messagescache tier.@@ -100,8 +89,7 @@ class OutputShaperSettings:verbosity_level: int = DEFAULT_VERBOSITY_LEVEL# False in ``mode="cache"``. Steering is the one lever that writes into the# provider prefix-cache key (it appends to the system-prompt tail, and on a- # body with no system field it creates one); effort routing and the- # thinking budget sit outside the key and stay on. See+ # body with no system field it creates one). See# :func:`steering_allowed_for`.steering_enabled: bool = True@@ -193,9 +181,7 @@ def steering_allowed_for(config: Any) -> bool:prefix-cache key byte-stable (see ``ProxyConfig.mode``). Verbosity steeringwrites into that key, so a block that CHANGES there trades a large, certaincache cost for a small, uncertain output saving — the wrong side of a- roughly 60x margin on a long context. Effort routing and the thinking- budget are unaffected: they ride request parameters outside the cache key,- so they keep saving in cache mode.+ roughly 60x margin on a long context.This is the gate for the sources that can change mid-conversation. It isNOT the last word: :func:`resolve_verbosity_level` lets an explicitlydiff --git a/headroom/proxy/server.py b/headroom/proxy/server.pyindex ca5d6d22f..310c19b42 100644--- a/headroom/proxy/server.py+++ b/headroom/proxy/server.py@@ -6520,7 +6520,7 @@ def run_server(sys.exit(1)# Seed the request-time coding-profile toggles (tool-search, dedupe, read- # protection, lossless→lossy, effort-router, block-char floor) into the+ # protection, lossless→lossy, block-char floor) into the# process env before serving, so downstream per-request readers pick them up.# Done here (not in the CLI command) so unit tests that mock run_server never# mutate os.environ. setdefault → explicit env still wins. MODE / profile arediff --git a/headroom/proxy/thinking_tokens.py b/headroom/proxy/thinking_tokens.pyindex ff1f5c531..3f04e5df2 100644--- a/headroom/proxy/thinking_tokens.py+++ b/headroom/proxy/thinking_tokens.py@@ -1,7 +1,7 @@"""Separate thinking tokens from visible output tokens.``output_tokens`` is a single number today, and it pools two quantities that-different levers move in opposite directions. Reasoning-effort routing cuts+different levers move in opposite directions. Thinking-budget levers cut*thinking*; verbosity steering cuts *visible text*. Summed into one counter,neither can be attributed: a request whose thinking dropped 4,000 tokens whileits prose grew 200 looks identical to one where nothing happened.diff --git a/scripts/eval_output_shaper.py b/scripts/eval_output_shaper.pyindex 3369de393..a6ff0c396 100644--- a/scripts/eval_output_shaper.py+++ b/scripts/eval_output_shaper.py@@ -8,11 +8,6 @@ includes thinking tokens.Scenario A (verbosity steering): a complex code-review ask. Baseline vsverbosity levels 2 and 3.-Scenario B (effort routing): an agentic transcript whose last message is a-clean tool_result (mechanical continuation) with `output_config.effort` set-to "xhigh" the way Claude Code pins it. The shaper lowers effort to "low"-for this turn only.-Usage:source .venv/bin/activate && python scripts/eval_output_shaper.pyRequires ANTHROPIC_API_KEY in the environment or in ./.env.@@ -106,59 +101,6 @@ def scenario_a_body() -> dict[str, Any]:}-def scenario_b_body() -> dict[str, Any]:- """Agentic mechanical continuation — exercises effort routing."""- return {- "model": MODEL,- "max_tokens": 8000,- "thinking": {"type": "adaptive"},- "output_config": {"effort": "xhigh"},- "system": (- "You are a coding agent. Use the Read tool to inspect files, then "- "report findings concisely."- ),- "tools": [- {- "name": "Read",- "description": "Read a file from the repository.",- "input_schema": {- "type": "object",- "properties": {"path": {"type": "string"}},- "required": ["path"],- },- }- ],- "messages": [- {- "role": "user",- "content": "Check whether cache.py has thread-safety issues.",- },- {- "role": "assistant",- "content": [- {"type": "text", "text": "Reading cache.py first."},- {- "type": "tool_use",- "id": "toolu_eval_01",- "name": "Read",- "input": {"path": "cache.py"},- },- ],- },- {- "role": "user",- "content": [- {- "type": "tool_result",- "tool_use_id": "toolu_eval_01",- "content": BUGGY_CODE,- }- ],- },- ],- }--def run(client: anthropic.Anthropic, body: dict[str, Any]) -> dict[str, int]:# The installed SDK may predate output_config as a typed kwarg; the API# accepts it either way, so pass it through extra_body.@@ -193,14 +135,6 @@ def main() -> int:shape_request(body, OutputShaperSettings(enabled=True, verbosity_level=level))conditions.append(("A:verbosity", f"shaped L{level}", body))- if which in ("B", "ALL"):- # Scenario B: baseline (effort=xhigh) vs shaped (effort routed to low).- conditions.append(("B:effort-routing", "baseline xhigh", scenario_b_body()))- body = scenario_b_body()- result = shape_request(body, OutputShaperSettings(enabled=True, verbosity_level=0))- assert body["output_config"]["effort"] == "low", result.labels- conditions.append(("B:effort-routing", "shaped low", body))-print(f"model={MODEL} trials={TRIALS}\n")print(f"{'scenario':<18} {'condition':<16} {'trial':<6} {'in_tok':>7} {'out_tok':>8}")print("-" * 60)