Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(tokenizers): don't tokenize image blocks as text in TiktokenCounter (#2093)
Abhay Singhae10d6c99d7b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b0b5bbd6b..f161cf555 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -102,6 +102,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Bug Fixes+* **tokenizers:** stop `TiktokenCounter.count_messages` from exploding on non-text content blocks. Its multi-part branch handled only `text` and OpenAI `image_url`; every other shape (Anthropic `image`/`tool_result`/`tool_use`, Strands blocks) fell through to `count_text(str(part))`, which json-stringified the base64 payload and tokenized it as text — a 1MB image counted as ~330K phantom tokens (~218x overcount in a standalone repro), corrupting every downstream budgeting/compression decision for multimodal OpenAI-model requests. Unknown block shapes now delegate to the base `_count_content_parts`, which prices images/documents by a bounded estimate (the overcount that helper already exists to prevent).* **install:** don't let a host env export override the manifest in persistent-docker deployments. `build_runtime_command` emitted the manifest's pinned `--env NAME=VALUE` pairs and then, for every host var matching a passthrough prefix, a bare `--env NAME`. Docker resolves duplicate `--env` last-wins, so a stale host export (e.g. `HEADROOM_BACKEND=anyllm`) that shared a passthrough prefix with a pinned manifest value (`HEADROOM_BACKEND=anthropic`) was appended after it and silently won, diverging the container from its deployment config. The bare passthrough is now skipped for any name the manifest already pins.* **memory:** honor explicit `store=false` on OpenAI `/v1/responses` requests by skipping Headroom memory-tool injection that depends on stored-response continuations. Memory context injection stays available, and requests no longer get rewritten to `store=true` behind the client's back ([#1944](https://github.com/headroomlabs-ai/headroom/issues/1944)).* **proxy/batch:** stop corrupting Google `batchGenerateContent` requests whose contents interleave text turns with text-less entries (functionCall/functionResponse/images). The batch handler restored preserved (non-text) entries with the raw-index loop that #836 replaced everywhere else — indexing the shorter `optimized_contents` (text-less entries produce no message) by the original `contents[]` index, which overwrites the wrong entry and drops any preserved entry whose original index is past the optimized length. A request like `[user text, model functionCall, user functionResponse, model text]` was forwarded to Google as two entries: the model's answer overwritten by the functionCall and the functionResponse dropped. The batch handler now uses the shared `_rebuild_gemini_contents` interleaving helper, so all entries survive in order.diff --git a/headroom/tokenizers/tiktoken_counter.py b/headroom/tokenizers/tiktoken_counter.pyindex 1a093e2df..728a098cf 100644--- a/headroom/tokenizers/tiktoken_counter.py+++ b/headroom/tokenizers/tiktoken_counter.py@@ -282,7 +282,15 @@ class TiktokenCounter(BaseTokenizer):else:total += 170 # Base for high detailelse:- total += self.count_text(str(part))+ # Any other block shape (Anthropic+ # image/tool_result/tool_use, Strands blocks)+ # is priced by the base handler, which uses a+ # bounded per-image/document estimate. Stringifying+ # it here would json-serialize a base64 blob and+ # count it as text — a 1MB image becomes ~330K+ # phantom tokens (the exact overcount base.py+ # _count_content_parts exists to prevent).+ total += self._count_content_parts([part])elif isinstance(part, str):total += self.count_text(part)elif key == "role":diff --git a/tests/test_tokenizers.py b/tests/test_tokenizers.pyindex 31714e8be..b0b6bcfe9 100644--- a/tests/test_tokenizers.py+++ b/tests/test_tokenizers.py@@ -103,6 +103,43 @@ class TestTiktokenCounter:count = counter.count_messages(messages)assert count > 0+ def test_count_messages_image_block_is_not_stringified(self):+ """An Anthropic-style image block must be priced as an image, not text.++ Over the wire the image arrives as a base64 string inside list content.+ The old count_messages else-branch stringified any non-text/non-image_url+ part and tokenized it as text, so a 1MB image counted as ~330K phantom+ tokens. The base handler prices image blocks by a bounded estimate, so the+ count must stay small regardless of the base64 payload size.+ """+ import base64++ counter = TiktokenCounter()+ blob = base64.b64encode(b"\x89PNG\r\n\x1a\n" + b"\x00" * 200_000).decode()+ messages = [+ {+ "role": "user",+ "content": [+ {"type": "text", "text": "What is in this screenshot?"},+ {+ "type": "image",+ "source": {+ "type": "base64",+ "media_type": "image/png",+ "data": blob,+ },+ },+ ],+ }+ ]++ count = counter.count_messages(messages)++ # The base64 blob alone would be tens of thousands of text tokens; a+ # bounded image estimate keeps the whole message well under that.+ assert count < 5000, count+ assert count < len(blob) // 10+def test_encode_decode_roundtrip(self):"""Test encode/decode roundtrip."""counter = TiktokenCounter()