Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(subscription): keep efficiency_pct from exceeding 100% (#2121)
Abhay Singh5fb449e90be3
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e6589a60d..fa0d41f1b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -130,6 +130,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0* **tokenizers:** resolve HuggingFace tokenizer names by the most-specific prefix. `get_tokenizer_name` scanned `MODEL_TO_TOKENIZER` in dict-insertion order and returned the first key the model merely starts with, so a short family key shadowed a more-specific one — `qwen2-7b-instruct` matched `qwen` before `qwen2`/`qwen2-7b` and resolved to the Qwen1 tokenizer (a different vocabulary, hence wrong token counts); `qwen2.5-*` and `deepseek-v2.x` were mis-resolved the same way. It now picks the longest matching prefix, mirroring the order-dependent-prefix guard the sibling tiktoken `get_encoding_for_model` already documents.* **pricing:** map retired `claude-3-sonnet-20240229` to a Sonnet-tier price instead of Haiku. When LiteLLM's cost DB lacks the retired model, resolution falls through to `MODEL_ALIASES`, which pointed Claude 3 Sonnet (a $3/$15-per-1M model) at `claude-3-haiku-20240307` ($0.25/$1.25) — a different tier that underpriced every cost/savings figure for that model ~12x on both input and output. It now aliases to `claude-sonnet-4-20250514`, the same-price target the other retired-Sonnet aliases already use.* **cache/semantic:** don't evict an unrelated entry when re-storing a key that is already cached. `SemanticCache.put` ran its at-capacity eviction loop before computing the entry's key, so overwriting a key that was already present (a duplicate or retried store) still evicted the LRU-oldest distinct entry even though an in-place update grows nothing. That silently dropped a live entry and turned a later lookup for it into a false cache miss. The key is now computed first and the eviction loop only runs when the key is genuinely new (mirroring `CompressionCache.store_compressed`, which deletes-then-inserts).+* **subscription:** cap `HeadroomContribution.efficiency_pct` at a real removal ratio so it can't exceed 100%. The numerator used `total_saved()` (which includes `tokens_saved_cache_reads`) while the denominator `raw_without_headroom()` excludes cache reads, so a contribution with large prefix-cache reads and small forwarded input reported impossible values (e.g. `tokens_submitted=100`, `tokens_saved_cache_reads=1000` → `1000.0%` on the dashboard). Cache reads are a provider-side discount on tokens that were still forwarded, not tokens Headroom removed, so the ratio now uses the sibling `compression_saved()` (compression + CLI filtering) as the numerator — bounded by its own denominator. `total_saved()` is unchanged for its other callers.* **proxy/anthropic:** cache the response under the same messages it was looked up by. The non-streaming `/v1/messages` path snapshots the scalar cache-key fields (system, tools, etc.) once before upstream to avoid post-mutation key drift (#327), but `messages` — the primary key component — was passed live at both `cache.get` and `cache.set`. Between the two, `messages` is reassigned by the enterprise security scan, the `pre_compress` hook, and image compression, so when any of those fired the response was stored under a different key than it was read by: the response cache never hit and accumulated unreachable entries until eviction. The lookup messages are now snapshotted alongside the other key fields and reused verbatim at `cache.set`.* **tokenizers:** stop `TiktokenCounter.count_messages` from exploding on non-text content blocks. Its multi-part branch handled only `text` and OpenAI `image_url`; every other shape (Anthropic `image`/`tool_result`/`tool_use`, Strands blocks) fell through to `count_text(str(part))`, which json-stringified the base64 payload and tokenized it as text — a 1MB image counted as ~330K phantom tokens (~218x overcount in a standalone repro), corrupting every downstream budgeting/compression decision for multimodal OpenAI-model requests. Unknown block shapes now delegate to the base `_count_content_parts`, which prices images/documents by a bounded estimate (the overcount that helper already exists to prevent).* **install:** don't let a host env export override the manifest in persistent-docker deployments. `build_runtime_command` emitted the manifest's pinned `--env NAME=VALUE` pairs and then, for every host var matching a passthrough prefix, a bare `--env NAME`. Docker resolves duplicate `--env` last-wins, so a stale host export (e.g. `HEADROOM_BACKEND=anyllm`) that shared a passthrough prefix with a pinned manifest value (`HEADROOM_BACKEND=anthropic`) was appended after it and silently won, diverging the container from its deployment config. The bare passthrough is now skipped for any name the manifest already pins.diff --git a/headroom/subscription/models.py b/headroom/subscription/models.pyindex 0ded25a6b..574b49dbb 100644--- a/headroom/subscription/models.py+++ b/headroom/subscription/models.py@@ -417,10 +417,18 @@ class HeadroomContribution:return self.tokens_submitted + self.tokens_saved_compression + self.cli_filtering_saved()def efficiency_pct(self) -> float:+ # Fraction of the pre-Headroom input that compression + CLI filtering+ # removed. Use compression_saved() (which excludes cache reads) as the+ # numerator: raw_without_headroom() also excludes cache reads, so mixing+ # total_saved() (which adds tokens_saved_cache_reads) into this ratio+ # made it inconsistent with its own denominator and let efficiency exceed+ # 100% (e.g. submitted=100, cache_reads=1000 -> 1000%). Cache reads are a+ # provider-side discount on tokens that were still forwarded, not tokens+ # Headroom removed, so they do not belong in a removal-efficiency ratio.raw = self.raw_without_headroom()if raw == 0:return 0.0- return round(self.total_saved() / raw * 100, 1)+ return round(self.compression_saved() / raw * 100, 1)def to_dict(self) -> dict[str, Any]:return {diff --git a/tests/test_subscription_contribution.py b/tests/test_subscription_contribution.pynew file mode 100644index 000000000..02b967628--- /dev/null+++ b/tests/test_subscription_contribution.py@@ -0,0 +1,37 @@+"""HeadroomContribution ratio methods."""++from __future__ import annotations++from headroom.subscription.models import HeadroomContribution+++def test_efficiency_pct_never_exceeds_100_from_cache_reads():+ """Cache reads must not push efficiency above 100%.++ Cache-read tokens are a provider-side discount on tokens that were still+ forwarded, not tokens Headroom removed. Counting them in the numerator while+ the denominator excludes them let efficiency report impossible values like+ 1000%.+ """+ c = HeadroomContribution(tokens_submitted=100, tokens_saved_cache_reads=1000)++ assert c.efficiency_pct() <= 100.0+ # Nothing was compressed or filtered, so the removal efficiency is 0.+ assert c.efficiency_pct() == 0.0+++def test_efficiency_pct_is_compression_removal_ratio():+ """Efficiency is compression + CLI filtering over the pre-Headroom input."""+ c = HeadroomContribution(+ tokens_submitted=1000,+ tokens_saved_compression=400,+ tokens_saved_cache_reads=300, # must not affect the ratio+ )++ # 400 removed out of (1000 forwarded + 400 removed) = 28.6%.+ assert c.efficiency_pct() == 28.6+ assert c.efficiency_pct() <= 100.0+++def test_efficiency_pct_zero_when_no_input():+ assert HeadroomContribution().efficiency_pct() == 0.0