Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(proxy/anthropic): scope session id by top-level system prompt (#2070)

Abhay Singhec6e60ea3ef2
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ddcf906ce..a6823eb91 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -102,6 +102,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Bug Fixes
* **ccr:** don't crash `parse_tool_call` on a CCR tool call whose arguments aren't an object. For the OpenAI/`openai_responses` shape the arguments are `json.loads`-decoded and only `JSONDecodeError` was caught, so a model that emitted `arguments='[]'`/`'"abc"'`/`'123'` (decoding to a list/str/number) — or a non-dict Anthropic `input` — reached `input_data.get("hash")` and raised `AttributeError`; a null `arguments` raised an uncaught `TypeError` from `json.loads(None)`. Both are now handled: the decode also catches `TypeError`, and a non-dict `input_data` returns `None` (not a valid CCR call) instead of crashing CCR response processing.
+* **proxy/anthropic:** give each Anthropic conversation its own session id. `SessionTrackerStore.compute_session_id` derived its fallback id from `model` + system text harvested only from `role:"system"` entries inside `messages` — but Anthropic carries the system prompt as a top-level `body["system"]` field, so genuine Anthropic requests (which never carry `x-headroom-session-id`) collapsed to `md5(model:[])` and every conversation on the same model shared one `PrefixCacheTracker`. That let session-sticky state cross-contaminate: conversation A's sticky `headroom_retrieve`/memory tools and `anthropic-beta` headers were injected into conversation B, and frozen-prefix/compression-cache state mixed across conversations. The Anthropic handler now folds the top-level `system` into the session-id inputs (prepending a synthetic `role:"system"` message used only to derive the id), giving distinct conversations distinct ids.
* **cache/semantic:** key entries by the full-context hash, not the trailing query text. `SemanticCache.put` stored each response under `sha256(query)[:16]` where `query` is only the last user message, and the exact-match branch of `get` returned the slot without checking the stored entry's `messages_hash`. Two requests that share a trailing message ("continue", "yes", "run the tests") but differ in earlier context therefore collided on one slot — the second overwrote the first, and the first's hash then resolved to the second's cached response (wrong data served). Entries are now keyed by `messages_hash` when present, and `get` verifies `entry.messages_hash` before returning.
* **proxy/openai:** stop PRE_SEND from reintroducing `tools: []` after the direct #728 fix. The OpenAI request handler now mirrors the existing `tools or _original_tools is not None` body-write guard during PRE_SEND write-back, so providers that reject empty tool arrays no longer see a tools field when the client omitted it, while explicit client `tools: []` remains preserved ([#1983](https://github.com/headroomlabs-ai/headroom/issues/1983)).
* **proxy/openai:** keep the exact Responses function name `terminal` resident during OpenAI tool-search deferral so cache-mode optimization stops forwarding `terminal.terminal` and triggering the reserved-namespace 400 on Codex Responses ([#1946](https://github.com/headroomlabs-ai/headroom/issues/1946)).
diff --git a/headroom/cache/prefix_tracker.py b/headroom/cache/prefix_tracker.py
index c30865395..585427447 100644
--- a/headroom/cache/prefix_tracker.py
+++ b/headroom/cache/prefix_tracker.py
@@ -806,6 +806,14 @@ class SessionTrackerStore:
Priority:
1. x-headroom-session-id header (explicit)
2. Hash of (model + system prompt) — stable per conversation
+
+ The system prompt is harvested from ``role:"system"`` entries in
+ ``messages``. Anthropic carries the system prompt as a top-level
+ ``body["system"]`` field instead, so its handler prepends that as a
+ synthetic ``role:"system"`` message before calling this — otherwise every
+ Anthropic conversation on the same model would collapse to one session id
+ and their session-sticky state (CCR/memory tools, beta headers, frozen
+ prefix) would cross-contaminate.
"""
# Check for explicit session header
if hasattr(request, "headers"):
diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.py
index a6b0f5fb4..9ee4878a1 100644
--- a/headroom/proxy/handlers/anthropic.py
+++ b/headroom/proxy/handlers/anthropic.py
@@ -1048,8 +1048,22 @@ class AnthropicHandlerMixin:
optimized_messages = messages
optimized_tokens = original_tokens
- # Get prefix cache tracker for this session
- session_id = self.session_tracker_store.compute_session_id(request, model, messages)
+ # Get prefix cache tracker for this session. Anthropic carries the
+ # system prompt as a top-level field, not a role:"system" message, so
+ # fold it into the session-id inputs as a synthetic system message —
+ # otherwise every conversation on the same model would share one
+ # session id (and its sticky CCR/memory tools, beta headers, and
+ # frozen-prefix state). This synthetic message only derives the id; it
+ # is never forwarded.
+ system_prompt = body.get("system")
+ session_messages = (
+ [{"role": "system", "content": system_prompt}, *messages]
+ if system_prompt is not None
+ else messages
+ )
+ session_id = self.session_tracker_store.compute_session_id(
+ request, model, session_messages
+ )
prefix_tracker = self.session_tracker_store.get_or_create(session_id, "anthropic")
frozen_message_count = prefix_tracker.get_frozen_message_count()
# Idle gap since the previous turn's response, snapshotted at fetch
diff --git a/tests/test_cache/test_prefix_tracker.py b/tests/test_cache/test_prefix_tracker.py
index 01d296755..76af68bdd 100644
--- a/tests/test_cache/test_prefix_tracker.py
+++ b/tests/test_cache/test_prefix_tracker.py
@@ -393,6 +393,38 @@ class TestSessionTrackerStore:
assert id_a != id_b
+ def test_compute_session_id_distinguishes_top_level_system(self, store):
+ """Anthropic carries the system prompt as a top-level field (not a
+ role:'system' message). The handler folds it in as a synthetic system
+ message so two conversations with the same model and turns but different
+ system prompts get distinct ids — otherwise they share one tracker and
+ their sticky state cross-contaminates. This exercises that mechanism."""
+
+ class MockRequest:
+ headers = {}
+
+ turns = [{"role": "user", "content": "hello"}]
+
+ def with_system(system):
+ # Mirror what handlers/anthropic.py does for the top-level system.
+ return [{"role": "system", "content": system}, *turns]
+
+ id_a = store.compute_session_id(
+ MockRequest(), "claude-3", with_system("You are a Python expert.")
+ )
+ id_b = store.compute_session_id(
+ MockRequest(), "claude-3", with_system("You are a Rust expert.")
+ )
+ assert id_a != id_b
+
+ # A list-of-text-blocks system folds the same text as the string form.
+ id_a_list = store.compute_session_id(
+ MockRequest(),
+ "claude-3",
+ with_system([{"type": "text", "text": "You are a Python expert."}]),
+ )
+ assert id_a_list == id_a
+
def test_compute_session_id_is_stable_when_only_non_system_turns_change(self, store):
"""Appending non-system turns should keep the same fallback session id."""