Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(subscription): keep efficiency_pct from exceeding 100% (#2121)

Abhay Singh5fb449e90be3
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e6589a60d..fa0d41f1b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -130,6 +130,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
* **tokenizers:** resolve HuggingFace tokenizer names by the most-specific prefix. `get_tokenizer_name` scanned `MODEL_TO_TOKENIZER` in dict-insertion order and returned the first key the model merely starts with, so a short family key shadowed a more-specific one — `qwen2-7b-instruct` matched `qwen` before `qwen2`/`qwen2-7b` and resolved to the Qwen1 tokenizer (a different vocabulary, hence wrong token counts); `qwen2.5-*` and `deepseek-v2.x` were mis-resolved the same way. It now picks the longest matching prefix, mirroring the order-dependent-prefix guard the sibling tiktoken `get_encoding_for_model` already documents.
* **pricing:** map retired `claude-3-sonnet-20240229` to a Sonnet-tier price instead of Haiku. When LiteLLM's cost DB lacks the retired model, resolution falls through to `MODEL_ALIASES`, which pointed Claude 3 Sonnet (a $3/$15-per-1M model) at `claude-3-haiku-20240307` ($0.25/$1.25) — a different tier that underpriced every cost/savings figure for that model ~12x on both input and output. It now aliases to `claude-sonnet-4-20250514`, the same-price target the other retired-Sonnet aliases already use.
* **cache/semantic:** don't evict an unrelated entry when re-storing a key that is already cached. `SemanticCache.put` ran its at-capacity eviction loop before computing the entry's key, so overwriting a key that was already present (a duplicate or retried store) still evicted the LRU-oldest distinct entry even though an in-place update grows nothing. That silently dropped a live entry and turned a later lookup for it into a false cache miss. The key is now computed first and the eviction loop only runs when the key is genuinely new (mirroring `CompressionCache.store_compressed`, which deletes-then-inserts).
+* **subscription:** cap `HeadroomContribution.efficiency_pct` at a real removal ratio so it can't exceed 100%. The numerator used `total_saved()` (which includes `tokens_saved_cache_reads`) while the denominator `raw_without_headroom()` excludes cache reads, so a contribution with large prefix-cache reads and small forwarded input reported impossible values (e.g. `tokens_submitted=100`, `tokens_saved_cache_reads=1000` → `1000.0%` on the dashboard). Cache reads are a provider-side discount on tokens that were still forwarded, not tokens Headroom removed, so the ratio now uses the sibling `compression_saved()` (compression + CLI filtering) as the numerator — bounded by its own denominator. `total_saved()` is unchanged for its other callers.
* **proxy/anthropic:** cache the response under the same messages it was looked up by. The non-streaming `/v1/messages` path snapshots the scalar cache-key fields (system, tools, etc.) once before upstream to avoid post-mutation key drift (#327), but `messages` — the primary key component — was passed live at both `cache.get` and `cache.set`. Between the two, `messages` is reassigned by the enterprise security scan, the `pre_compress` hook, and image compression, so when any of those fired the response was stored under a different key than it was read by: the response cache never hit and accumulated unreachable entries until eviction. The lookup messages are now snapshotted alongside the other key fields and reused verbatim at `cache.set`.
* **tokenizers:** stop `TiktokenCounter.count_messages` from exploding on non-text content blocks. Its multi-part branch handled only `text` and OpenAI `image_url`; every other shape (Anthropic `image`/`tool_result`/`tool_use`, Strands blocks) fell through to `count_text(str(part))`, which json-stringified the base64 payload and tokenized it as text — a 1MB image counted as ~330K phantom tokens (~218x overcount in a standalone repro), corrupting every downstream budgeting/compression decision for multimodal OpenAI-model requests. Unknown block shapes now delegate to the base `_count_content_parts`, which prices images/documents by a bounded estimate (the overcount that helper already exists to prevent).
* **install:** don't let a host env export override the manifest in persistent-docker deployments. `build_runtime_command` emitted the manifest's pinned `--env NAME=VALUE` pairs and then, for every host var matching a passthrough prefix, a bare `--env NAME`. Docker resolves duplicate `--env` last-wins, so a stale host export (e.g. `HEADROOM_BACKEND=anyllm`) that shared a passthrough prefix with a pinned manifest value (`HEADROOM_BACKEND=anthropic`) was appended after it and silently won, diverging the container from its deployment config. The bare passthrough is now skipped for any name the manifest already pins.
diff --git a/headroom/subscription/models.py b/headroom/subscription/models.py
index 0ded25a6b..574b49dbb 100644
--- a/headroom/subscription/models.py
+++ b/headroom/subscription/models.py
@@ -417,10 +417,18 @@ class HeadroomContribution:
return self.tokens_submitted + self.tokens_saved_compression + self.cli_filtering_saved()
def efficiency_pct(self) -> float:
+ # Fraction of the pre-Headroom input that compression + CLI filtering
+ # removed. Use compression_saved() (which excludes cache reads) as the
+ # numerator: raw_without_headroom() also excludes cache reads, so mixing
+ # total_saved() (which adds tokens_saved_cache_reads) into this ratio
+ # made it inconsistent with its own denominator and let efficiency exceed
+ # 100% (e.g. submitted=100, cache_reads=1000 -> 1000%). Cache reads are a
+ # provider-side discount on tokens that were still forwarded, not tokens
+ # Headroom removed, so they do not belong in a removal-efficiency ratio.
raw = self.raw_without_headroom()
if raw == 0:
return 0.0
- return round(self.total_saved() / raw * 100, 1)
+ return round(self.compression_saved() / raw * 100, 1)
def to_dict(self) -> dict[str, Any]:
return {
diff --git a/tests/test_subscription_contribution.py b/tests/test_subscription_contribution.py
new file mode 100644
index 000000000..02b967628
--- /dev/null
+++ b/tests/test_subscription_contribution.py
@@ -0,0 +1,37 @@
+"""HeadroomContribution ratio methods."""
+
+from __future__ import annotations
+
+from headroom.subscription.models import HeadroomContribution
+
+
+def test_efficiency_pct_never_exceeds_100_from_cache_reads():
+ """Cache reads must not push efficiency above 100%.
+
+ Cache-read tokens are a provider-side discount on tokens that were still
+ forwarded, not tokens Headroom removed. Counting them in the numerator while
+ the denominator excludes them let efficiency report impossible values like
+ 1000%.
+ """
+ c = HeadroomContribution(tokens_submitted=100, tokens_saved_cache_reads=1000)
+
+ assert c.efficiency_pct() <= 100.0
+ # Nothing was compressed or filtered, so the removal efficiency is 0.
+ assert c.efficiency_pct() == 0.0
+
+
+def test_efficiency_pct_is_compression_removal_ratio():
+ """Efficiency is compression + CLI filtering over the pre-Headroom input."""
+ c = HeadroomContribution(
+ tokens_submitted=1000,
+ tokens_saved_compression=400,
+ tokens_saved_cache_reads=300, # must not affect the ratio
+ )
+
+ # 400 removed out of (1000 forwarded + 400 removed) = 28.6%.
+ assert c.efficiency_pct() == 28.6
+ assert c.efficiency_pct() <= 100.0
+
+
+def test_efficiency_pct_zero_when_no_input():
+ assert HeadroomContribution().efficiency_pct() == 0.0