Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(cache/semantic): key entries by context hash, not query text (#2022)

Abhay Singhd8783ab89b49
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1f09828ab..2726bb43e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -32,6 +32,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Bug Fixes
+* **cache/semantic:** key entries by the full-context hash, not the trailing query text. `SemanticCache.put` stored each response under `sha256(query)[:16]` where `query` is only the last user message, and the exact-match branch of `get` returned the slot without checking the stored entry's `messages_hash`. Two requests that share a trailing message ("continue", "yes", "run the tests") but differ in earlier context therefore collided on one slot — the second overwrote the first, and the first's hash then resolved to the second's cached response (wrong data served). Entries are now keyed by `messages_hash` when present, and `get` verifies `entry.messages_hash` before returning.
* **proxy/openai:** stop PRE_SEND from reintroducing `tools: []` after the direct #728 fix. The OpenAI request handler now mirrors the existing `tools or _original_tools is not None` body-write guard during PRE_SEND write-back, so providers that reject empty tool arrays no longer see a tools field when the client omitted it, while explicit client `tools: []` remains preserved ([#1983](https://github.com/headroomlabs-ai/headroom/issues/1983)).
* **proxy/openai:** keep the exact Responses function name `terminal` resident during OpenAI tool-search deferral so cache-mode optimization stops forwarding `terminal.terminal` and triggering the reserved-namespace 400 on Codex Responses ([#1946](https://github.com/headroomlabs-ai/headroom/issues/1946)).
* **proxy/openai:** thread the savings-profile kwargs into the live `/v1/chat/completions` compression path. The chat handler called `openai_pipeline.apply()` without `proxy_pipeline_kwargs(config)`, so `HEADROOM_SAVINGS_PROFILE=agent-90` (and the individual `compress_user_messages`/`target_ratio`/`min_tokens_to_compress`/... knobs) were silently dropped — OpenAI-compatible clients like OpenCode kept protecting user messages and missed the configured profile. Both the token-mode and non-token chat branches now pass the profile kwargs, matching `handlers/anthropic.py` and the dedicated OpenAI compress endpoint ([#1534](https://github.com/headroomlabs-ai/headroom/issues/1534)).
diff --git a/headroom/cache/semantic.py b/headroom/cache/semantic.py
index 4a6904602..bcf968a03 100644
--- a/headroom/cache/semantic.py
+++ b/headroom/cache/semantic.py
@@ -152,9 +152,13 @@ class SemanticCache:
key = self._hash_index.get(messages_hash)
if key and key in self._cache:
entry = self._cache[key]
- self._touch(key)
- self._hits += 1
- return entry
+ # Verify the stored entry really belongs to this request. Guards
+ # against a stale index mapping ever pointing at an entry that was
+ # overwritten by a different conversation sharing the same key.
+ if entry.messages_hash == messages_hash:
+ self._touch(key)
+ self._hits += 1
+ return entry
# Try semantic similarity if we have embedding function
if self._embedding_fn:
@@ -197,8 +201,12 @@ class SemanticCache:
if self._embedding_fn:
embedding = self._embedding_fn(query)
- # Create cache key
- key = self._generate_key(query)
+ # Create cache key. Prefer the full-context hash: two requests that share
+ # a trailing user message ("continue", "yes", "run the tests") but differ
+ # in earlier context must NOT collide on one query-derived slot and
+ # overwrite each other. Fall back to the query hash only when no
+ # messages_hash is supplied (e.g. embedding-only usage).
+ key = messages_hash or self._generate_key(query)
now = time.time()
entry = CacheEntry(
diff --git a/tests/test_cache/test_semantic.py b/tests/test_cache/test_semantic.py
index dc8b11d91..8cbb15da9 100644
--- a/tests/test_cache/test_semantic.py
+++ b/tests/test_cache/test_semantic.py
@@ -51,6 +51,27 @@ class TestSemanticCache:
entry = cache.get("Unknown query", messages_hash="unknown")
assert entry is None
+ def test_same_query_different_context_does_not_collide(self, cache):
+ """Two requests that share a trailing user message but differ in earlier
+ context (distinct messages_hash) must not overwrite each other. Before the
+ fix both were keyed by sha256(query), so the second clobbered the first and
+ the first's hash resolved to the second's response."""
+ cache.put("run the tests", {"text": "response A"}, messages_hash="ctxA")
+ cache.put("run the tests", {"text": "response B"}, messages_hash="ctxB")
+
+ got_a = cache.get("run the tests", messages_hash="ctxA")
+ got_b = cache.get("run the tests", messages_hash="ctxB")
+
+ assert got_a is not None and got_a.response == {"text": "response A"}
+ assert got_b is not None and got_b.response == {"text": "response B"}
+
+ def test_exact_match_verifies_messages_hash(self, cache):
+ """A stored entry is only returned when its messages_hash matches the
+ looked-up hash — never another conversation's cached response."""
+ cache.put("continue", {"text": "A"}, messages_hash="hA")
+ # A lookup for a hash that isn't stored is a miss, not a wrong hit.
+ assert cache.get("continue", messages_hash="hB") is None
+
def test_lru_eviction(self):
"""Test LRU eviction when at capacity."""
config = SemanticCacheConfig(max_entries=3)