Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(tokenizers): recurse into list-content tool_result blocks (#2081)
Abhay Singhdfb1d37ed619
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex bc92f84d1..da5e62d23 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -102,6 +102,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Bug Fixes+* **tokenizers:** recurse into a native `tool_result` whose content is a list of blocks instead of JSON-serializing it. `_count_content_parts` counted a `tool_result` with list content via `_count_serialized` (json.dumps + sample), so a base64 image nested in a tool result (computer-use / MCP screenshot tools) was priced as text — a ~50-200x overcount (a ~200KB screenshot read as ~70K tokens instead of ~1600). It now recurses into the nested blocks, matching the sibling Strands `toolResult` branch, so the image is priced structurally. The overcount made a single screenshot appear to blow past the model's context window and triggered unnecessary/over-aggressive compression.* **tokenizers:** price dense scripts (CJK/Kana/Hangul) in the fixed-ratio estimator path. `EstimatingTokenCounter.count_text` applied the CJK correction only on the auto-detect path; its fixed-ratio early return divided by the Latin ratio with no adjustment. The registry builds every provider-calibrated counter with a fixed ratio (Anthropic 3.5, Google 4.0, Cohere 4.0, Moonshot 3.1), and the Anthropic/Gemini proxy handlers count via `get_tokenizer(model).count_messages`, so a CJK-heavy context read as ~40-55% of its true token size — it could fall under the size/backpressure gates and skip compression, and every `x-headroom-tokens-before` metric for CJK traffic was materially wrong. The fixed-ratio path now applies the same dense-script split.* **ccr:** don't crash `parse_tool_call` on a CCR tool call whose arguments aren't an object. For the OpenAI/`openai_responses` shape the arguments are `json.loads`-decoded and only `JSONDecodeError` was caught, so a model that emitted `arguments='[]'`/`'"abc"'`/`'123'` (decoding to a list/str/number) — or a non-dict Anthropic `input` — reached `input_data.get("hash")` and raised `AttributeError`; a null `arguments` raised an uncaught `TypeError` from `json.loads(None)`. Both are now handled: the decode also catches `TypeError`, and a non-dict `input_data` returns `None` (not a valid CCR call) instead of crashing CCR response processing.* **mcp/codex:** don't corrupt an unparseable or non-table `config.toml` on register. `CodexRegistrar.register_server` only guarded against clobbering a user-managed entry when `get_server` returned one, but `get_server` returns `None` both for an unparseable TOML file and for an `mcp_servers`/`mcp_servers.<name>` that is present but not a table. In those cases `register_server` fell through to `_write_block`, which blindly appended a `[mcp_servers.<name>]` table — appending into an unparseable file, or creating a duplicate `[mcp_servers.headroom]` key alongside a non-table entry (e.g. `headroom = "..."`), which `tomllib`/codex then reject, destroying a previously-valid config. It now refuses (`FAILED`) and leaves the file untouched, mirroring the claude (#1660) and opencode (#1661) guards.diff --git a/headroom/tokenizers/base.py b/headroom/tokenizers/base.pyindex da105f7ca..e90c5534c 100644--- a/headroom/tokenizers/base.py+++ b/headroom/tokenizers/base.py@@ -159,6 +159,13 @@ class BaseTokenizer(ABC):content = part.get("content", "")if isinstance(content, str):total += self.count_text(content)+ elif isinstance(content, list):+ # Recurse into nested blocks (matching the Strands+ # toolResult branch below). A tool that returns an image+ # nests a base64 block here; serializing it would price+ # the ~KB/MB base64 string as text — a 50-200x overcount+ # (a screenshot reads as tens of thousands of tokens).+ total += self._count_content_parts(content)else:total += self._count_serialized(content)elif part_type == "tool_use":diff --git a/tests/test_tokenizers.py b/tests/test_tokenizers.pyindex 332696343..31714e8be 100644--- a/tests/test_tokenizers.py+++ b/tests/test_tokenizers.py@@ -613,6 +613,38 @@ class TestLargeToolBlobEstimation:exact = tok.count_text(json.dumps(big))assert abs(tok._count_serialized(big) - exact) / exact < 0.10+ def test_tool_result_list_recurses_into_image_block(self):+ """A tool that returns an image nests a base64 block inside a+ `tool_result` list. Serializing it prices the base64 as text (a 50-200x+ overcount); recursing into the block prices the image at ~1600 tokens."""+ tok = EstimatingTokenCounter()+ messages = [+ {+ "role": "user",+ "content": [+ {+ "type": "tool_result",+ "tool_use_id": "t1",+ "content": [+ {+ "type": "image",+ "source": {+ "type": "base64",+ "media_type": "image/png",+ "data": "A" * 280_000, # ~280KB base64+ },+ }+ ],+ }+ ],+ }+ ]++ count = tok.count_messages(messages)++ # The image is priced structurally (~1600), not as a huge text blob.+ assert count < 5_000+def test_oversized_estimate_never_overcounts(self):"""R4 (prefer false negatives): a token-dense head + sparse tail must notover-count. Counting per leaf cannot extrapolate a dense front slice to the