Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
Fix image token estimate: use 1600 tokens (pixel-based), not 85
chopratejasf40dc2fbf984
diff --git a/headroom/tokenizers/base.py b/headroom/tokenizers/base.pyindex 3b449c0ac..a0b782044 100644--- a/headroom/tokenizers/base.py+++ b/headroom/tokenizers/base.py@@ -133,11 +133,13 @@ class BaseTokenizer(ABC):if part_type == "text":total += self.count_text(part.get("text", ""))elif part_type in ("image_url", "image"):- # Images have fixed token cost — NOT tokenized as text.- # Anthropic images are type="image" with base64 in source.data.- # Without this, the base64 string gets json.dumps'd and counted- # as text tokens (1MB image = ~330K fake tokens).- total += 85 # Base image token count+ # Images are NOT tokenized as text — they have a pixel-based cost.+ # Anthropic: tokens = (width * height) / 750, max ~1600 after resize.+ # OpenAI: similar tile-based calculation, ~765 tokens for high-detail.+ # We use 1600 as a conservative estimate (max after auto-resize).+ # This prevents the base64 blob from being json.dumps'd and counted+ # as text tokens (1MB image = ~330K fake tokens without this).+ total += 1600elif part_type == "tool_result":content = part.get("content", "")if isinstance(content, str):