Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(proxy/anthropic): scope session id by top-level system prompt (#2070)
Abhay Singhec6e60ea3ef2
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ddcf906ce..a6823eb91 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -102,6 +102,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Bug Fixes* **ccr:** don't crash `parse_tool_call` on a CCR tool call whose arguments aren't an object. For the OpenAI/`openai_responses` shape the arguments are `json.loads`-decoded and only `JSONDecodeError` was caught, so a model that emitted `arguments='[]'`/`'"abc"'`/`'123'` (decoding to a list/str/number) — or a non-dict Anthropic `input` — reached `input_data.get("hash")` and raised `AttributeError`; a null `arguments` raised an uncaught `TypeError` from `json.loads(None)`. Both are now handled: the decode also catches `TypeError`, and a non-dict `input_data` returns `None` (not a valid CCR call) instead of crashing CCR response processing.+* **proxy/anthropic:** give each Anthropic conversation its own session id. `SessionTrackerStore.compute_session_id` derived its fallback id from `model` + system text harvested only from `role:"system"` entries inside `messages` — but Anthropic carries the system prompt as a top-level `body["system"]` field, so genuine Anthropic requests (which never carry `x-headroom-session-id`) collapsed to `md5(model:[])` and every conversation on the same model shared one `PrefixCacheTracker`. That let session-sticky state cross-contaminate: conversation A's sticky `headroom_retrieve`/memory tools and `anthropic-beta` headers were injected into conversation B, and frozen-prefix/compression-cache state mixed across conversations. The Anthropic handler now folds the top-level `system` into the session-id inputs (prepending a synthetic `role:"system"` message used only to derive the id), giving distinct conversations distinct ids.* **cache/semantic:** key entries by the full-context hash, not the trailing query text. `SemanticCache.put` stored each response under `sha256(query)[:16]` where `query` is only the last user message, and the exact-match branch of `get` returned the slot without checking the stored entry's `messages_hash`. Two requests that share a trailing message ("continue", "yes", "run the tests") but differ in earlier context therefore collided on one slot — the second overwrote the first, and the first's hash then resolved to the second's cached response (wrong data served). Entries are now keyed by `messages_hash` when present, and `get` verifies `entry.messages_hash` before returning.* **proxy/openai:** stop PRE_SEND from reintroducing `tools: []` after the direct #728 fix. The OpenAI request handler now mirrors the existing `tools or _original_tools is not None` body-write guard during PRE_SEND write-back, so providers that reject empty tool arrays no longer see a tools field when the client omitted it, while explicit client `tools: []` remains preserved ([#1983](https://github.com/headroomlabs-ai/headroom/issues/1983)).* **proxy/openai:** keep the exact Responses function name `terminal` resident during OpenAI tool-search deferral so cache-mode optimization stops forwarding `terminal.terminal` and triggering the reserved-namespace 400 on Codex Responses ([#1946](https://github.com/headroomlabs-ai/headroom/issues/1946)).diff --git a/headroom/cache/prefix_tracker.py b/headroom/cache/prefix_tracker.pyindex c30865395..585427447 100644--- a/headroom/cache/prefix_tracker.py+++ b/headroom/cache/prefix_tracker.py@@ -806,6 +806,14 @@ class SessionTrackerStore:Priority:1. x-headroom-session-id header (explicit)2. Hash of (model + system prompt) — stable per conversation++ The system prompt is harvested from ``role:"system"`` entries in+ ``messages``. Anthropic carries the system prompt as a top-level+ ``body["system"]`` field instead, so its handler prepends that as a+ synthetic ``role:"system"`` message before calling this — otherwise every+ Anthropic conversation on the same model would collapse to one session id+ and their session-sticky state (CCR/memory tools, beta headers, frozen+ prefix) would cross-contaminate."""# Check for explicit session headerif hasattr(request, "headers"):diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.pyindex a6b0f5fb4..9ee4878a1 100644--- a/headroom/proxy/handlers/anthropic.py+++ b/headroom/proxy/handlers/anthropic.py@@ -1048,8 +1048,22 @@ class AnthropicHandlerMixin:optimized_messages = messagesoptimized_tokens = original_tokens- # Get prefix cache tracker for this session- session_id = self.session_tracker_store.compute_session_id(request, model, messages)+ # Get prefix cache tracker for this session. Anthropic carries the+ # system prompt as a top-level field, not a role:"system" message, so+ # fold it into the session-id inputs as a synthetic system message —+ # otherwise every conversation on the same model would share one+ # session id (and its sticky CCR/memory tools, beta headers, and+ # frozen-prefix state). This synthetic message only derives the id; it+ # is never forwarded.+ system_prompt = body.get("system")+ session_messages = (+ [{"role": "system", "content": system_prompt}, *messages]+ if system_prompt is not None+ else messages+ )+ session_id = self.session_tracker_store.compute_session_id(+ request, model, session_messages+ )prefix_tracker = self.session_tracker_store.get_or_create(session_id, "anthropic")frozen_message_count = prefix_tracker.get_frozen_message_count()# Idle gap since the previous turn's response, snapshotted at fetchdiff --git a/tests/test_cache/test_prefix_tracker.py b/tests/test_cache/test_prefix_tracker.pyindex 01d296755..76af68bdd 100644--- a/tests/test_cache/test_prefix_tracker.py+++ b/tests/test_cache/test_prefix_tracker.py@@ -393,6 +393,38 @@ class TestSessionTrackerStore:assert id_a != id_b+ def test_compute_session_id_distinguishes_top_level_system(self, store):+ """Anthropic carries the system prompt as a top-level field (not a+ role:'system' message). The handler folds it in as a synthetic system+ message so two conversations with the same model and turns but different+ system prompts get distinct ids — otherwise they share one tracker and+ their sticky state cross-contaminates. This exercises that mechanism."""++ class MockRequest:+ headers = {}++ turns = [{"role": "user", "content": "hello"}]++ def with_system(system):+ # Mirror what handlers/anthropic.py does for the top-level system.+ return [{"role": "system", "content": system}, *turns]++ id_a = store.compute_session_id(+ MockRequest(), "claude-3", with_system("You are a Python expert.")+ )+ id_b = store.compute_session_id(+ MockRequest(), "claude-3", with_system("You are a Rust expert.")+ )+ assert id_a != id_b++ # A list-of-text-blocks system folds the same text as the string form.+ id_a_list = store.compute_session_id(+ MockRequest(),+ "claude-3",+ with_system([{"type": "text", "text": "You are a Python expert."}]),+ )+ assert id_a_list == id_a+def test_compute_session_id_is_stable_when_only_non_system_turns_change(self, store):"""Appending non-system turns should keep the same fallback session id."""