Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(proxy/anthropic): cache response under the looked-up messages (#327) (#2124)
Abhay Singhdbb4e4cf4809
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 636da84a7..d1292d08b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -119,6 +119,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0* **tokenizers:** resolve HuggingFace tokenizer names by the most-specific prefix. `get_tokenizer_name` scanned `MODEL_TO_TOKENIZER` in dict-insertion order and returned the first key the model merely starts with, so a short family key shadowed a more-specific one — `qwen2-7b-instruct` matched `qwen` before `qwen2`/`qwen2-7b` and resolved to the Qwen1 tokenizer (a different vocabulary, hence wrong token counts); `qwen2.5-*` and `deepseek-v2.x` were mis-resolved the same way. It now picks the longest matching prefix, mirroring the order-dependent-prefix guard the sibling tiktoken `get_encoding_for_model` already documents.* **pricing:** map retired `claude-3-sonnet-20240229` to a Sonnet-tier price instead of Haiku. When LiteLLM's cost DB lacks the retired model, resolution falls through to `MODEL_ALIASES`, which pointed Claude 3 Sonnet (a $3/$15-per-1M model) at `claude-3-haiku-20240307` ($0.25/$1.25) — a different tier that underpriced every cost/savings figure for that model ~12x on both input and output. It now aliases to `claude-sonnet-4-20250514`, the same-price target the other retired-Sonnet aliases already use.* **cache/semantic:** don't evict an unrelated entry when re-storing a key that is already cached. `SemanticCache.put` ran its at-capacity eviction loop before computing the entry's key, so overwriting a key that was already present (a duplicate or retried store) still evicted the LRU-oldest distinct entry even though an in-place update grows nothing. That silently dropped a live entry and turned a later lookup for it into a false cache miss. The key is now computed first and the eviction loop only runs when the key is genuinely new (mirroring `CompressionCache.store_compressed`, which deletes-then-inserts).+* **proxy/anthropic:** cache the response under the same messages it was looked up by. The non-streaming `/v1/messages` path snapshots the scalar cache-key fields (system, tools, etc.) once before upstream to avoid post-mutation key drift (#327), but `messages` — the primary key component — was passed live at both `cache.get` and `cache.set`. Between the two, `messages` is reassigned by the enterprise security scan, the `pre_compress` hook, and image compression, so when any of those fired the response was stored under a different key than it was read by: the response cache never hit and accumulated unreachable entries until eviction. The lookup messages are now snapshotted alongside the other key fields and reused verbatim at `cache.set`.* **tokenizers:** stop `TiktokenCounter.count_messages` from exploding on non-text content blocks. Its multi-part branch handled only `text` and OpenAI `image_url`; every other shape (Anthropic `image`/`tool_result`/`tool_use`, Strands blocks) fell through to `count_text(str(part))`, which json-stringified the base64 payload and tokenized it as text — a 1MB image counted as ~330K phantom tokens (~218x overcount in a standalone repro), corrupting every downstream budgeting/compression decision for multimodal OpenAI-model requests. Unknown block shapes now delegate to the base `_count_content_parts`, which prices images/documents by a bounded estimate (the overcount that helper already exists to prevent).* **install:** don't let a host env export override the manifest in persistent-docker deployments. `build_runtime_command` emitted the manifest's pinned `--env NAME=VALUE` pairs and then, for every host var matching a passthrough prefix, a bare `--env NAME`. Docker resolves duplicate `--env` last-wins, so a stale host export (e.g. `HEADROOM_BACKEND=anyllm`) that shared a passthrough prefix with a pinned manifest value (`HEADROOM_BACKEND=anthropic`) was appended after it and silently won, diverging the container from its deployment config. The bare passthrough is now skipped for any name the manifest already pins.* **memory:** honor explicit `store=false` on OpenAI `/v1/responses` requests by skipping Headroom memory-tool injection that depends on stored-response continuations. Memory context injection stays available, and requests no longer get rewritten to `store=true` behind the client's back ([#1944](https://github.com/headroomlabs-ai/headroom/issues/1944)).diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.pyindex 8afdc6049..fcb752e5b 100644--- a/headroom/proxy/handlers/anthropic.py+++ b/headroom/proxy/handlers/anthropic.py@@ -930,6 +930,14 @@ class AnthropicHandlerMixin:"thinking": body.get("thinking"),"output_config": body.get("output_config"),}+ # Snapshot the lookup messages too. `messages` is the primary cache+ # key component, but it is reassigned below by the security scan, the+ # pre_compress hook, and image compression, so caching the response+ # under the live `messages` would store it under a different key than+ # it was looked up by — the cache would never hit and would fill with+ # unreachable entries. Reuse this raw snapshot verbatim at cache.set+ # (the same reason cache_key_fields is snapshotted here, #327).+ cache_lookup_messages = messages# Check cache (non-streaming only)cache_hit = Falseif self.cache and not stream:@@ -3057,10 +3065,12 @@ class AnthropicHandlerMixin:original_messages=next_original_messages,)- # Cache response+ # Cache response under the SAME key it was looked up by:+ # cache_lookup_messages is the raw pre-mutation snapshot, not+ # the live (compressed/hooked) `messages` (#327).if self.cache and response.status_code == 200:await self.cache.set(- messages,+ cache_lookup_messages,model,response.content,dict(response.headers),diff --git a/tests/test_anthropic_pre_upstream_backpressure.py b/tests/test_anthropic_pre_upstream_backpressure.pyindex c0327a004..c2fc9db72 100644--- a/tests/test_anthropic_pre_upstream_backpressure.py+++ b/tests/test_anthropic_pre_upstream_backpressure.py@@ -21,6 +21,7 @@ Covers:from __future__ import annotationsimport asyncio+import copyimport jsonimport loggingimport os@@ -1005,3 +1006,68 @@ def test_security_scan_still_returns_200_for_ok_upstream():response = anyio.run(handler.handle_anthropic_messages, request)assert response.status_code == 200+++# --------------------------------------------------------------------------- #+# Response cache must key on the looked-up messages, not the mutated ones #+# --------------------------------------------------------------------------- #+++class _RecordingCache:+ """Records the messages passed to get() and set() so the test can assert the+ response is cached under the same key it was looked up by."""++ def __init__(self) -> None:+ self.get_messages = None+ self.set_messages = None++ async def get(self, messages, model, **fields):+ self.get_messages = copy.deepcopy(messages)+ return None # force a miss so the upstream response gets cached++ async def set(self, messages, model, content, headers, **kwargs):+ self.set_messages = copy.deepcopy(messages)+++class _MutatingSecurity:+ """Stands in for an enterprise-security scanner that rewrites (anonymizes)+ the request messages, reproducing the get -> mutate -> set hazard."""++ def scan_request(self, messages, ctx):+ mutated = [dict(m, content="MUTATED") for m in messages]+ return mutated, {"anonymization": {}}++ def scan_response(self, resp_json, ctx):+ return resp_json+++def test_response_cache_keys_on_lookup_messages_not_mutated():+ """The response must be cached under the same messages it was looked up by.++ `messages` is reassigned after the cache.get (here by the security scan), so+ caching under the live `messages` stores the entry under a different key than+ it was read by -- the cache never hits and fills with unreachable entries.+ """+ sem = asyncio.Semaphore(2)+ handler = _DummyAnthropicHandler(anthropic_pre_upstream_sem=sem)+ cache = _RecordingCache()+ handler.cache = cache+ handler.security = _MutatingSecurity()++ request = _build_request(+ {+ "model": "claude-3-5-sonnet-latest",+ "messages": [{"role": "user", "content": "hello"}],+ },+ {"authorization": "Bearer sk-ant-api-test"},+ )++ with _tokenizer_patch():+ anyio.run(handler.handle_anthropic_messages, request)++ assert cache.get_messages is not None, "cache.get was not called"+ assert cache.set_messages is not None, "cache.set was not called"+ # Same messages at get and set -> same key -> the cache can actually hit.+ assert cache.set_messages == cache.get_messages+ # And specifically the raw lookup messages, not the scanner's rewrite.+ assert cache.set_messages == [{"role": "user", "content": "hello"}]