Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(tokenizers): price CJK in the fixed-ratio estimator path (#2080)

Abhay Singhcd3d5aa10c5f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0f59388e8..0e1e3ee17 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -101,6 +101,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Bug Fixes
+* **tokenizers:** price dense scripts (CJK/Kana/Hangul) in the fixed-ratio estimator path. `EstimatingTokenCounter.count_text` applied the CJK correction only on the auto-detect path; its fixed-ratio early return divided by the Latin ratio with no adjustment. The registry builds every provider-calibrated counter with a fixed ratio (Anthropic 3.5, Google 4.0, Cohere 4.0, Moonshot 3.1), and the Anthropic/Gemini proxy handlers count via `get_tokenizer(model).count_messages`, so a CJK-heavy context read as ~40-55% of its true token size — it could fall under the size/backpressure gates and skip compression, and every `x-headroom-tokens-before` metric for CJK traffic was materially wrong. The fixed-ratio path now applies the same dense-script split.
* **ccr:** don't crash `parse_tool_call` on a CCR tool call whose arguments aren't an object. For the OpenAI/`openai_responses` shape the arguments are `json.loads`-decoded and only `JSONDecodeError` was caught, so a model that emitted `arguments='[]'`/`'"abc"'`/`'123'` (decoding to a list/str/number) — or a non-dict Anthropic `input` — reached `input_data.get("hash")` and raised `AttributeError`; a null `arguments` raised an uncaught `TypeError` from `json.loads(None)`. Both are now handled: the decode also catches `TypeError`, and a non-dict `input_data` returns `None` (not a valid CCR call) instead of crashing CCR response processing.
* **proxy/vertex:** route Vertex `publisher=google` (Gemini) requests to the region matching the request path. `vertex_generate_content`, `vertex_stream_generate_content`, and `vertex_count_tokens` discarded the path's `location` and forwarded to the single fixed host from `_api_target(proxy, "vertex")` (default `us-central1`), instead of the region-aware `_vertex_target_for_location` the sibling Anthropic `rawPredict` route already uses. So a request to `.../locations/europe-west1/publishers/google/...` was sent to a `us-central1` host, which Vertex rejects on the region/host mismatch. The three google routes now derive the host from the request's `location` (operator-pinned upstreams are still honored).
* **proxy/anthropic:** give each Anthropic conversation its own session id. `SessionTrackerStore.compute_session_id` derived its fallback id from `model` + system text harvested only from `role:"system"` entries inside `messages` — but Anthropic carries the system prompt as a top-level `body["system"]` field, so genuine Anthropic requests (which never carry `x-headroom-session-id`) collapsed to `md5(model:[])` and every conversation on the same model shared one `PrefixCacheTracker`. That let session-sticky state cross-contaminate: conversation A's sticky `headroom_retrieve`/memory tools and `anthropic-beta` headers were injected into conversation B, and frozen-prefix/compression-cache state mixed across conversations. The Anthropic handler now folds the top-level `system` into the session-id inputs (prepending a synthetic `role:"system"` message used only to derive the id), giving distinct conversations distinct ids.
diff --git a/headroom/tokenizers/estimator.py b/headroom/tokenizers/estimator.py
index a28fc12d6..e20ea15d5 100644
--- a/headroom/tokenizers/estimator.py
+++ b/headroom/tokenizers/estimator.py
@@ -100,9 +100,19 @@ class EstimatingTokenCounter(BaseTokenizer):
if not text:
return 0
- # Use fixed ratio if provided
+ # Use fixed ratio if provided. Dense scripts (CJK/Kana/Hangul) still
+ # tokenize at ~1 token per character, so pricing them at the (Latin)
+ # fixed ratio under-counts by 2-4x — the same correction the auto path
+ # below applies. The registry builds every provider-calibrated counter
+ # (Anthropic 3.5, Google 4.0, Cohere 4.0, Moonshot 3.1) with a fixed
+ # ratio, so this is the live proxy count path for those providers.
if self._fixed_ratio is not None:
- return max(1, int(len(text) / self._fixed_ratio + 0.5))
+ cjk_chars = self._count_cjk_chars(text)
+ other_chars = len(text) - cjk_chars
+ return max(
+ 1,
+ int(other_chars / self._fixed_ratio + cjk_chars / self.CHARS_PER_TOKEN_CJK + 0.5),
+ )
# Auto-detect content type and adjust ratio
ratio = self._detect_ratio(text)
diff --git a/tests/test_tokenizers.py b/tests/test_tokenizers.py
index 371a084f6..332696343 100644
--- a/tests/test_tokenizers.py
+++ b/tests/test_tokenizers.py
@@ -175,6 +175,26 @@ class TestEstimatingTokenCounter:
count = counter.count_text(text)
assert count == 10 # 50 / 5 = 10
+ def test_count_text_fixed_ratio_prices_cjk(self):
+ """The fixed-ratio path must price dense scripts (CJK) like the auto path.
+
+ The registry builds provider counters (Anthropic 3.5, Google 4.0, ...)
+ with a fixed ratio; before the fix CJK was priced at the Latin ratio, so
+ a large CJK context read as ~40-55% of its true size and could skip
+ compression."""
+ counter = EstimatingTokenCounter(chars_per_token=3.5)
+ cjk = "これはテストです" * 100 # 800 dense-script chars, no ASCII
+
+ count = counter.count_text(cjk)
+
+ # ~1 token per 1.5 chars (CHARS_PER_TOKEN_CJK), not 1 per 3.5.
+ assert count == pytest.approx(len(cjk) / 1.5, rel=0.05)
+ # Far higher than the old Latin-ratio estimate.
+ assert count > len(cjk) / 3.5 * 2
+
+ # ASCII is unaffected by the fixed ratio.
+ assert counter.count_text("x" * 35) == 10
+
def test_count_text_minimum_one(self):
"""Test minimum of 1 token."""
counter = EstimatingTokenCounter()