Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(tokenizers): recurse into list-content tool_result blocks (#2081)

Abhay Singhdfb1d37ed619
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bc92f84d1..da5e62d23 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -102,6 +102,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Bug Fixes
+* **tokenizers:** recurse into a native `tool_result` whose content is a list of blocks instead of JSON-serializing it. `_count_content_parts` counted a `tool_result` with list content via `_count_serialized` (json.dumps + sample), so a base64 image nested in a tool result (computer-use / MCP screenshot tools) was priced as text — a ~50-200x overcount (a ~200KB screenshot read as ~70K tokens instead of ~1600). It now recurses into the nested blocks, matching the sibling Strands `toolResult` branch, so the image is priced structurally. The overcount made a single screenshot appear to blow past the model's context window and triggered unnecessary/over-aggressive compression.
* **tokenizers:** price dense scripts (CJK/Kana/Hangul) in the fixed-ratio estimator path. `EstimatingTokenCounter.count_text` applied the CJK correction only on the auto-detect path; its fixed-ratio early return divided by the Latin ratio with no adjustment. The registry builds every provider-calibrated counter with a fixed ratio (Anthropic 3.5, Google 4.0, Cohere 4.0, Moonshot 3.1), and the Anthropic/Gemini proxy handlers count via `get_tokenizer(model).count_messages`, so a CJK-heavy context read as ~40-55% of its true token size — it could fall under the size/backpressure gates and skip compression, and every `x-headroom-tokens-before` metric for CJK traffic was materially wrong. The fixed-ratio path now applies the same dense-script split.
* **ccr:** don't crash `parse_tool_call` on a CCR tool call whose arguments aren't an object. For the OpenAI/`openai_responses` shape the arguments are `json.loads`-decoded and only `JSONDecodeError` was caught, so a model that emitted `arguments='[]'`/`'"abc"'`/`'123'` (decoding to a list/str/number) — or a non-dict Anthropic `input` — reached `input_data.get("hash")` and raised `AttributeError`; a null `arguments` raised an uncaught `TypeError` from `json.loads(None)`. Both are now handled: the decode also catches `TypeError`, and a non-dict `input_data` returns `None` (not a valid CCR call) instead of crashing CCR response processing.
* **mcp/codex:** don't corrupt an unparseable or non-table `config.toml` on register. `CodexRegistrar.register_server` only guarded against clobbering a user-managed entry when `get_server` returned one, but `get_server` returns `None` both for an unparseable TOML file and for an `mcp_servers`/`mcp_servers.<name>` that is present but not a table. In those cases `register_server` fell through to `_write_block`, which blindly appended a `[mcp_servers.<name>]` table — appending into an unparseable file, or creating a duplicate `[mcp_servers.headroom]` key alongside a non-table entry (e.g. `headroom = "..."`), which `tomllib`/codex then reject, destroying a previously-valid config. It now refuses (`FAILED`) and leaves the file untouched, mirroring the claude (#1660) and opencode (#1661) guards.
diff --git a/headroom/tokenizers/base.py b/headroom/tokenizers/base.py
index da105f7ca..e90c5534c 100644
--- a/headroom/tokenizers/base.py
+++ b/headroom/tokenizers/base.py
@@ -159,6 +159,13 @@ class BaseTokenizer(ABC):
content = part.get("content", "")
if isinstance(content, str):
total += self.count_text(content)
+ elif isinstance(content, list):
+ # Recurse into nested blocks (matching the Strands
+ # toolResult branch below). A tool that returns an image
+ # nests a base64 block here; serializing it would price
+ # the ~KB/MB base64 string as text — a 50-200x overcount
+ # (a screenshot reads as tens of thousands of tokens).
+ total += self._count_content_parts(content)
else:
total += self._count_serialized(content)
elif part_type == "tool_use":
diff --git a/tests/test_tokenizers.py b/tests/test_tokenizers.py
index 332696343..31714e8be 100644
--- a/tests/test_tokenizers.py
+++ b/tests/test_tokenizers.py
@@ -613,6 +613,38 @@ class TestLargeToolBlobEstimation:
exact = tok.count_text(json.dumps(big))
assert abs(tok._count_serialized(big) - exact) / exact < 0.10
+ def test_tool_result_list_recurses_into_image_block(self):
+ """A tool that returns an image nests a base64 block inside a
+ `tool_result` list. Serializing it prices the base64 as text (a 50-200x
+ overcount); recursing into the block prices the image at ~1600 tokens."""
+ tok = EstimatingTokenCounter()
+ messages = [
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_use_id": "t1",
+ "content": [
+ {
+ "type": "image",
+ "source": {
+ "type": "base64",
+ "media_type": "image/png",
+ "data": "A" * 280_000, # ~280KB base64
+ },
+ }
+ ],
+ }
+ ],
+ }
+ ]
+
+ count = tok.count_messages(messages)
+
+ # The image is priced structurally (~1600), not as a huge text blob.
+ assert count < 5_000
+
def test_oversized_estimate_never_overcounts(self):
"""R4 (prefer false negatives): a token-dense head + sparse tail must not
over-count. Counting per leaf cannot extrapolate a dense front slice to the