Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(tokenizers): price CJK in the fixed-ratio estimator path (#2080)
Abhay Singhcd3d5aa10c5f
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 0f59388e8..0e1e3ee17 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -101,6 +101,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Bug Fixes+* **tokenizers:** price dense scripts (CJK/Kana/Hangul) in the fixed-ratio estimator path. `EstimatingTokenCounter.count_text` applied the CJK correction only on the auto-detect path; its fixed-ratio early return divided by the Latin ratio with no adjustment. The registry builds every provider-calibrated counter with a fixed ratio (Anthropic 3.5, Google 4.0, Cohere 4.0, Moonshot 3.1), and the Anthropic/Gemini proxy handlers count via `get_tokenizer(model).count_messages`, so a CJK-heavy context read as ~40-55% of its true token size — it could fall under the size/backpressure gates and skip compression, and every `x-headroom-tokens-before` metric for CJK traffic was materially wrong. The fixed-ratio path now applies the same dense-script split.* **ccr:** don't crash `parse_tool_call` on a CCR tool call whose arguments aren't an object. For the OpenAI/`openai_responses` shape the arguments are `json.loads`-decoded and only `JSONDecodeError` was caught, so a model that emitted `arguments='[]'`/`'"abc"'`/`'123'` (decoding to a list/str/number) — or a non-dict Anthropic `input` — reached `input_data.get("hash")` and raised `AttributeError`; a null `arguments` raised an uncaught `TypeError` from `json.loads(None)`. Both are now handled: the decode also catches `TypeError`, and a non-dict `input_data` returns `None` (not a valid CCR call) instead of crashing CCR response processing.* **proxy/vertex:** route Vertex `publisher=google` (Gemini) requests to the region matching the request path. `vertex_generate_content`, `vertex_stream_generate_content`, and `vertex_count_tokens` discarded the path's `location` and forwarded to the single fixed host from `_api_target(proxy, "vertex")` (default `us-central1`), instead of the region-aware `_vertex_target_for_location` the sibling Anthropic `rawPredict` route already uses. So a request to `.../locations/europe-west1/publishers/google/...` was sent to a `us-central1` host, which Vertex rejects on the region/host mismatch. The three google routes now derive the host from the request's `location` (operator-pinned upstreams are still honored).* **proxy/anthropic:** give each Anthropic conversation its own session id. `SessionTrackerStore.compute_session_id` derived its fallback id from `model` + system text harvested only from `role:"system"` entries inside `messages` — but Anthropic carries the system prompt as a top-level `body["system"]` field, so genuine Anthropic requests (which never carry `x-headroom-session-id`) collapsed to `md5(model:[])` and every conversation on the same model shared one `PrefixCacheTracker`. That let session-sticky state cross-contaminate: conversation A's sticky `headroom_retrieve`/memory tools and `anthropic-beta` headers were injected into conversation B, and frozen-prefix/compression-cache state mixed across conversations. The Anthropic handler now folds the top-level `system` into the session-id inputs (prepending a synthetic `role:"system"` message used only to derive the id), giving distinct conversations distinct ids.diff --git a/headroom/tokenizers/estimator.py b/headroom/tokenizers/estimator.pyindex a28fc12d6..e20ea15d5 100644--- a/headroom/tokenizers/estimator.py+++ b/headroom/tokenizers/estimator.py@@ -100,9 +100,19 @@ class EstimatingTokenCounter(BaseTokenizer):if not text:return 0- # Use fixed ratio if provided+ # Use fixed ratio if provided. Dense scripts (CJK/Kana/Hangul) still+ # tokenize at ~1 token per character, so pricing them at the (Latin)+ # fixed ratio under-counts by 2-4x — the same correction the auto path+ # below applies. The registry builds every provider-calibrated counter+ # (Anthropic 3.5, Google 4.0, Cohere 4.0, Moonshot 3.1) with a fixed+ # ratio, so this is the live proxy count path for those providers.if self._fixed_ratio is not None:- return max(1, int(len(text) / self._fixed_ratio + 0.5))+ cjk_chars = self._count_cjk_chars(text)+ other_chars = len(text) - cjk_chars+ return max(+ 1,+ int(other_chars / self._fixed_ratio + cjk_chars / self.CHARS_PER_TOKEN_CJK + 0.5),+ )# Auto-detect content type and adjust ratioratio = self._detect_ratio(text)diff --git a/tests/test_tokenizers.py b/tests/test_tokenizers.pyindex 371a084f6..332696343 100644--- a/tests/test_tokenizers.py+++ b/tests/test_tokenizers.py@@ -175,6 +175,26 @@ class TestEstimatingTokenCounter:count = counter.count_text(text)assert count == 10 # 50 / 5 = 10+ def test_count_text_fixed_ratio_prices_cjk(self):+ """The fixed-ratio path must price dense scripts (CJK) like the auto path.++ The registry builds provider counters (Anthropic 3.5, Google 4.0, ...)+ with a fixed ratio; before the fix CJK was priced at the Latin ratio, so+ a large CJK context read as ~40-55% of its true size and could skip+ compression."""+ counter = EstimatingTokenCounter(chars_per_token=3.5)+ cjk = "これはテストです" * 100 # 800 dense-script chars, no ASCII++ count = counter.count_text(cjk)++ # ~1 token per 1.5 chars (CHARS_PER_TOKEN_CJK), not 1 per 3.5.+ assert count == pytest.approx(len(cjk) / 1.5, rel=0.05)+ # Far higher than the old Latin-ratio estimate.+ assert count > len(cjk) / 3.5 * 2++ # ASCII is unaffected by the fixed ratio.+ assert counter.count_text("x" * 35) == 10+def test_count_text_minimum_one(self):"""Test minimum of 1 token."""counter = EstimatingTokenCounter()