Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(cache/semantic): key entries by context hash, not query text (#2022)
Abhay Singhd8783ab89b49
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1f09828ab..2726bb43e 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -32,6 +32,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Bug Fixes+* **cache/semantic:** key entries by the full-context hash, not the trailing query text. `SemanticCache.put` stored each response under `sha256(query)[:16]` where `query` is only the last user message, and the exact-match branch of `get` returned the slot without checking the stored entry's `messages_hash`. Two requests that share a trailing message ("continue", "yes", "run the tests") but differ in earlier context therefore collided on one slot — the second overwrote the first, and the first's hash then resolved to the second's cached response (wrong data served). Entries are now keyed by `messages_hash` when present, and `get` verifies `entry.messages_hash` before returning.* **proxy/openai:** stop PRE_SEND from reintroducing `tools: []` after the direct #728 fix. The OpenAI request handler now mirrors the existing `tools or _original_tools is not None` body-write guard during PRE_SEND write-back, so providers that reject empty tool arrays no longer see a tools field when the client omitted it, while explicit client `tools: []` remains preserved ([#1983](https://github.com/headroomlabs-ai/headroom/issues/1983)).* **proxy/openai:** keep the exact Responses function name `terminal` resident during OpenAI tool-search deferral so cache-mode optimization stops forwarding `terminal.terminal` and triggering the reserved-namespace 400 on Codex Responses ([#1946](https://github.com/headroomlabs-ai/headroom/issues/1946)).* **proxy/openai:** thread the savings-profile kwargs into the live `/v1/chat/completions` compression path. The chat handler called `openai_pipeline.apply()` without `proxy_pipeline_kwargs(config)`, so `HEADROOM_SAVINGS_PROFILE=agent-90` (and the individual `compress_user_messages`/`target_ratio`/`min_tokens_to_compress`/... knobs) were silently dropped — OpenAI-compatible clients like OpenCode kept protecting user messages and missed the configured profile. Both the token-mode and non-token chat branches now pass the profile kwargs, matching `handlers/anthropic.py` and the dedicated OpenAI compress endpoint ([#1534](https://github.com/headroomlabs-ai/headroom/issues/1534)).diff --git a/headroom/cache/semantic.py b/headroom/cache/semantic.pyindex 4a6904602..bcf968a03 100644--- a/headroom/cache/semantic.py+++ b/headroom/cache/semantic.py@@ -152,9 +152,13 @@ class SemanticCache:key = self._hash_index.get(messages_hash)if key and key in self._cache:entry = self._cache[key]- self._touch(key)- self._hits += 1- return entry+ # Verify the stored entry really belongs to this request. Guards+ # against a stale index mapping ever pointing at an entry that was+ # overwritten by a different conversation sharing the same key.+ if entry.messages_hash == messages_hash:+ self._touch(key)+ self._hits += 1+ return entry# Try semantic similarity if we have embedding functionif self._embedding_fn:@@ -197,8 +201,12 @@ class SemanticCache:if self._embedding_fn:embedding = self._embedding_fn(query)- # Create cache key- key = self._generate_key(query)+ # Create cache key. Prefer the full-context hash: two requests that share+ # a trailing user message ("continue", "yes", "run the tests") but differ+ # in earlier context must NOT collide on one query-derived slot and+ # overwrite each other. Fall back to the query hash only when no+ # messages_hash is supplied (e.g. embedding-only usage).+ key = messages_hash or self._generate_key(query)now = time.time()entry = CacheEntry(diff --git a/tests/test_cache/test_semantic.py b/tests/test_cache/test_semantic.pyindex dc8b11d91..8cbb15da9 100644--- a/tests/test_cache/test_semantic.py+++ b/tests/test_cache/test_semantic.py@@ -51,6 +51,27 @@ class TestSemanticCache:entry = cache.get("Unknown query", messages_hash="unknown")assert entry is None+ def test_same_query_different_context_does_not_collide(self, cache):+ """Two requests that share a trailing user message but differ in earlier+ context (distinct messages_hash) must not overwrite each other. Before the+ fix both were keyed by sha256(query), so the second clobbered the first and+ the first's hash resolved to the second's response."""+ cache.put("run the tests", {"text": "response A"}, messages_hash="ctxA")+ cache.put("run the tests", {"text": "response B"}, messages_hash="ctxB")++ got_a = cache.get("run the tests", messages_hash="ctxA")+ got_b = cache.get("run the tests", messages_hash="ctxB")++ assert got_a is not None and got_a.response == {"text": "response A"}+ assert got_b is not None and got_b.response == {"text": "response B"}++ def test_exact_match_verifies_messages_hash(self, cache):+ """A stored entry is only returned when its messages_hash matches the+ looked-up hash — never another conversation's cached response."""+ cache.put("continue", {"text": "A"}, messages_hash="hA")+ # A lookup for a hash that isn't stored is a miss, not a wrong hit.+ assert cache.get("continue", messages_hash="hB") is None+def test_lru_eviction(self):"""Test LRU eviction when at capacity."""config = SemanticCacheConfig(max_entries=3)