Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(proxy/anthropic): cache response under the looked-up messages (#327) (#2124)

Abhay Singhdbb4e4cf4809
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 636da84a7..d1292d08b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -119,6 +119,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
* **tokenizers:** resolve HuggingFace tokenizer names by the most-specific prefix. `get_tokenizer_name` scanned `MODEL_TO_TOKENIZER` in dict-insertion order and returned the first key the model merely starts with, so a short family key shadowed a more-specific one — `qwen2-7b-instruct` matched `qwen` before `qwen2`/`qwen2-7b` and resolved to the Qwen1 tokenizer (a different vocabulary, hence wrong token counts); `qwen2.5-*` and `deepseek-v2.x` were mis-resolved the same way. It now picks the longest matching prefix, mirroring the order-dependent-prefix guard the sibling tiktoken `get_encoding_for_model` already documents.
* **pricing:** map retired `claude-3-sonnet-20240229` to a Sonnet-tier price instead of Haiku. When LiteLLM's cost DB lacks the retired model, resolution falls through to `MODEL_ALIASES`, which pointed Claude 3 Sonnet (a $3/$15-per-1M model) at `claude-3-haiku-20240307` ($0.25/$1.25) — a different tier that underpriced every cost/savings figure for that model ~12x on both input and output. It now aliases to `claude-sonnet-4-20250514`, the same-price target the other retired-Sonnet aliases already use.
* **cache/semantic:** don't evict an unrelated entry when re-storing a key that is already cached. `SemanticCache.put` ran its at-capacity eviction loop before computing the entry's key, so overwriting a key that was already present (a duplicate or retried store) still evicted the LRU-oldest distinct entry even though an in-place update grows nothing. That silently dropped a live entry and turned a later lookup for it into a false cache miss. The key is now computed first and the eviction loop only runs when the key is genuinely new (mirroring `CompressionCache.store_compressed`, which deletes-then-inserts).
+* **proxy/anthropic:** cache the response under the same messages it was looked up by. The non-streaming `/v1/messages` path snapshots the scalar cache-key fields (system, tools, etc.) once before upstream to avoid post-mutation key drift (#327), but `messages` — the primary key component — was passed live at both `cache.get` and `cache.set`. Between the two, `messages` is reassigned by the enterprise security scan, the `pre_compress` hook, and image compression, so when any of those fired the response was stored under a different key than it was read by: the response cache never hit and accumulated unreachable entries until eviction. The lookup messages are now snapshotted alongside the other key fields and reused verbatim at `cache.set`.
* **tokenizers:** stop `TiktokenCounter.count_messages` from exploding on non-text content blocks. Its multi-part branch handled only `text` and OpenAI `image_url`; every other shape (Anthropic `image`/`tool_result`/`tool_use`, Strands blocks) fell through to `count_text(str(part))`, which json-stringified the base64 payload and tokenized it as text — a 1MB image counted as ~330K phantom tokens (~218x overcount in a standalone repro), corrupting every downstream budgeting/compression decision for multimodal OpenAI-model requests. Unknown block shapes now delegate to the base `_count_content_parts`, which prices images/documents by a bounded estimate (the overcount that helper already exists to prevent).
* **install:** don't let a host env export override the manifest in persistent-docker deployments. `build_runtime_command` emitted the manifest's pinned `--env NAME=VALUE` pairs and then, for every host var matching a passthrough prefix, a bare `--env NAME`. Docker resolves duplicate `--env` last-wins, so a stale host export (e.g. `HEADROOM_BACKEND=anyllm`) that shared a passthrough prefix with a pinned manifest value (`HEADROOM_BACKEND=anthropic`) was appended after it and silently won, diverging the container from its deployment config. The bare passthrough is now skipped for any name the manifest already pins.
* **memory:** honor explicit `store=false` on OpenAI `/v1/responses` requests by skipping Headroom memory-tool injection that depends on stored-response continuations. Memory context injection stays available, and requests no longer get rewritten to `store=true` behind the client's back ([#1944](https://github.com/headroomlabs-ai/headroom/issues/1944)).
diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.py
index 8afdc6049..fcb752e5b 100644
--- a/headroom/proxy/handlers/anthropic.py
+++ b/headroom/proxy/handlers/anthropic.py
@@ -930,6 +930,14 @@ class AnthropicHandlerMixin:
"thinking": body.get("thinking"),
"output_config": body.get("output_config"),
}
+ # Snapshot the lookup messages too. `messages` is the primary cache
+ # key component, but it is reassigned below by the security scan, the
+ # pre_compress hook, and image compression, so caching the response
+ # under the live `messages` would store it under a different key than
+ # it was looked up by — the cache would never hit and would fill with
+ # unreachable entries. Reuse this raw snapshot verbatim at cache.set
+ # (the same reason cache_key_fields is snapshotted here, #327).
+ cache_lookup_messages = messages
# Check cache (non-streaming only)
cache_hit = False
if self.cache and not stream:
@@ -3057,10 +3065,12 @@ class AnthropicHandlerMixin:
original_messages=next_original_messages,
)
- # Cache response
+ # Cache response under the SAME key it was looked up by:
+ # cache_lookup_messages is the raw pre-mutation snapshot, not
+ # the live (compressed/hooked) `messages` (#327).
if self.cache and response.status_code == 200:
await self.cache.set(
- messages,
+ cache_lookup_messages,
model,
response.content,
dict(response.headers),
diff --git a/tests/test_anthropic_pre_upstream_backpressure.py b/tests/test_anthropic_pre_upstream_backpressure.py
index c0327a004..c2fc9db72 100644
--- a/tests/test_anthropic_pre_upstream_backpressure.py
+++ b/tests/test_anthropic_pre_upstream_backpressure.py
@@ -21,6 +21,7 @@ Covers:
from __future__ import annotations
import asyncio
+import copy
import json
import logging
import os
@@ -1005,3 +1006,68 @@ def test_security_scan_still_returns_200_for_ok_upstream():
response = anyio.run(handler.handle_anthropic_messages, request)
assert response.status_code == 200
+
+
+# --------------------------------------------------------------------------- #
+# Response cache must key on the looked-up messages, not the mutated ones #
+# --------------------------------------------------------------------------- #
+
+
+class _RecordingCache:
+ """Records the messages passed to get() and set() so the test can assert the
+ response is cached under the same key it was looked up by."""
+
+ def __init__(self) -> None:
+ self.get_messages = None
+ self.set_messages = None
+
+ async def get(self, messages, model, **fields):
+ self.get_messages = copy.deepcopy(messages)
+ return None # force a miss so the upstream response gets cached
+
+ async def set(self, messages, model, content, headers, **kwargs):
+ self.set_messages = copy.deepcopy(messages)
+
+
+class _MutatingSecurity:
+ """Stands in for an enterprise-security scanner that rewrites (anonymizes)
+ the request messages, reproducing the get -> mutate -> set hazard."""
+
+ def scan_request(self, messages, ctx):
+ mutated = [dict(m, content="MUTATED") for m in messages]
+ return mutated, {"anonymization": {}}
+
+ def scan_response(self, resp_json, ctx):
+ return resp_json
+
+
+def test_response_cache_keys_on_lookup_messages_not_mutated():
+ """The response must be cached under the same messages it was looked up by.
+
+ `messages` is reassigned after the cache.get (here by the security scan), so
+ caching under the live `messages` stores the entry under a different key than
+ it was read by -- the cache never hits and fills with unreachable entries.
+ """
+ sem = asyncio.Semaphore(2)
+ handler = _DummyAnthropicHandler(anthropic_pre_upstream_sem=sem)
+ cache = _RecordingCache()
+ handler.cache = cache
+ handler.security = _MutatingSecurity()
+
+ request = _build_request(
+ {
+ "model": "claude-3-5-sonnet-latest",
+ "messages": [{"role": "user", "content": "hello"}],
+ },
+ {"authorization": "Bearer sk-ant-api-test"},
+ )
+
+ with _tokenizer_patch():
+ anyio.run(handler.handle_anthropic_messages, request)
+
+ assert cache.get_messages is not None, "cache.get was not called"
+ assert cache.set_messages is not None, "cache.set was not called"
+ # Same messages at get and set -> same key -> the cache can actually hit.
+ assert cache.set_messages == cache.get_messages
+ # And specifically the raw lookup messages, not the scanner's rewrite.
+ assert cache.set_messages == [{"role": "user", "content": "hello"}]