Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(cache): stable session identity and per-conversation prefix trackers under agentic clients (#2193)
Andrei Boldyrev7bfb1d7f38ac
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 388573cbd..63239d167 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- **install:** `headroom install apply --env KEY=VALUE` (repeatable) passes environment variables into supervised runners (macOS launchd, Linux systemd/cron, Windows services/tasks). These runners previously started with a bare environment and did not inherit the interactive shell's exports — e.g. a custom `HEADROOM_WORKSPACE_DIR` never reached the supervised process, so `headroom install agent run` looked for its manifest in the wrong location and failed outright even though `install apply` itself succeeded. `--env` values are merged into `DeploymentManifest.base_env` last, so they can override auto-derived defaults, and are threaded into the generated `run-headroom.sh`/`ensure-headroom.sh` (and Windows equivalents) as `export`/`$env:` lines before the `exec`.### Fixed+- **cache/prefix-freeze:** resolve `PrefixCacheTracker`s per conversation lineage within a session id, so concurrent conversations sharing a fallback id no longer thrash one tracker's frozen-prefix state ([#2085](https://github.com/headroomlabs-ai/headroom/issues/2085)). Without an `x-headroom-session-id` header the fallback id hashes `model + system prompt` — identical across a Claude Code session and every one of its parallel subagents (and any set of sessions reusing one system prompt). On the shared tracker their interleaved histories cross-contaminate the freeze state: the forwarded prefix is byte-unstable on nearly every turn and the provider prompt cache is re-written instead of read — reported as ~4.4x cache-creation inflation and a 2.5–3x net cost increase under Claude Code. `SessionTrackerStore.resolve_tracker` now reuses the tracker whose previous request messages are a prefix of the incoming history (client histories are append-only, so a conversation's next request always extends its previous one), starts a fresh lineage when the history diverges or was rewritten (client-side compaction — that provider cache line is gone anyway), and caps lineages per session id (`PrefixFreezeConfig.max_lineages_per_session`, default 32; over-cap conversations share one overflow tracker instead of evicting established lineages, so a fan-out storm past the cap degrades only its own tail — and `0` disables lineage splitting). Matching compares the original client bytes under the same canonical cross-turn equivalence as the cache-stable delta path (`_canonicalize_for_prefix_compare`), so a moved cache breakpoint, string<->block content sugar, or per-turn transport annotations do not read as a rewrite. Separately, the fallback id now hashes only the LEADING run of `role:"system"` messages: agentic clients interleave `<system-reminder>` turns into the history as actual system-role messages (hook output, skills lists, truncation notices), and hashing those rotated the session id mid-conversation — orphaning the prefix tracker and every other session-sticky subsystem (beta headers, CCR/memory registries, the compression cache) each time a reminder landed. Both handler paths now derive the session id and the lineage from the same original client bytes, so a turn-dependent hook rewrite cannot rotate one without the other. The session id itself never changes: session-sticky state keyed on it (beta-header stickiness, CCR and memory-tool registries, the compression cache) is untouched, and a single-conversation session keeps its exact previous behavior (the first lineage lives under the bare id).- **proxy/bedrock:** wire `PrefixCacheTracker` updates into both Bedrock backend paths (`handle_anthropic_messages`'s non-streaming branch in `anthropic.py`, and `_stream_response_bedrock` in `streaming.py`). `update_from_response()` was previously only called from the direct-Anthropic-API branch; both Bedrock branches returned before ever reaching it, so the tracker's state stayed permanently empty for the life of a session on any `--backend bedrock` deployment: `extract_cache_stable_delta()` always saw no previous turn, and `--mode cache` fell back to full unmodified passthrough on every turn instead of freezing the already-cached prefix and compressing only the new suffix.- **install:** `install_supervisor`'s macOS branch did an unconditional `launchctl bootout` followed by a bare `bootstrap` with no retry, unlike `start_supervisor`, which already rides out the ~15s EIO (error 5) window launchd exhibits for several seconds after a bootout. This left `install apply`'s own reinstall path (and anything that re-applies a deployment, e.g. a future `headroom doctor --fix`) exposed to a race that previously required manual recovery (bootout + remove the plist + reapply). Extracted the retry loop already used by `start_supervisor` into a shared `_bootstrap_with_retry()` helper, now used by both call sites.- **proxy/savings:** `SavingsTracker.record_request()` only appended a history point when `tokens_saved > 0` (headroom's own lossy compression). In `--mode cache`, `tokens_saved` is near-always 0 by design, since the frozen prefix is byte-replayed rather than compressed to keep the provider's prompt cache warm. That silently dropped every history point on a cache-mode deployment even when `cache_read_tokens`/`cache_savings_usd` were large, making `headroom-monthly`-style tooling read as a total savings collapse. The guard now fires on `tokens_saved` OR `cache_read_tokens`, and the appended entry carries `cache_read_tokens`/`cache_savings_usd` so downstream consumers can show them; `_normalize_history_entry` defaults both fields to 0/0.0 for legacy entries that predate this change.diff --git a/headroom/cache/prefix_tracker.py b/headroom/cache/prefix_tracker.pyindex 585427447..d7e62f436 100644--- a/headroom/cache/prefix_tracker.py+++ b/headroom/cache/prefix_tracker.py@@ -18,9 +18,11 @@ from __future__ import annotationsimport copyimport hashlib+import itertoolsimport jsonimport loggingimport time+from collections import OrderedDictfrom dataclasses import dataclassfrom typing import Any@@ -70,6 +72,12 @@ class PrefixFreezeConfig:# default in `_PROVIDER_CACHE_TTL_SECONDS`. Set to 3600 for a session that# uses Anthropic's 1h cache breakpoint so idle-gap attribution stays honest.cache_ttl_seconds: int | None = None+ # Cap on concurrent conversation lineages tracked per session id (#2085).+ # A fan-out storm (many parallel subagents sharing one fallback id) evicts+ # the shortest-chain lineage instead of growing without bound. Raise it+ # for workspaces that genuinely run more concurrent conversations on one+ # model + system prompt.+ max_lineages_per_session: int = 32@dataclass@@ -764,10 +772,33 @@ class PrefixCacheTracker:return counts+def _lineage_snapshot(obj: Any) -> Any:+ """Structural copy of a canonical projection for lineage-chain storage.++ Copies dict/list structure (immutable leaves are shared, so this costs+ structure, not bytes) to isolate the stored chain from downstream in-place+ mutation of the request, and normalizes NaN to a sentinel — ``json.loads``+ accepts bare ``NaN`` and ``NaN != NaN``, so a byte-identical resend would+ otherwise read as a rewritten history on every turn. Values inside opaque+ tool payloads are walked for copying only; no keys are dropped (see+ ``_OPAQUE_PAYLOAD_KEYS``).+ """+ if isinstance(obj, dict):+ return {k: _lineage_snapshot(v) for k, v in obj.items()}+ if isinstance(obj, list):+ return [_lineage_snapshot(v) for v in obj]+ if isinstance(obj, float) and obj != obj:+ return "\x00nan"+ return obj++class SessionTrackerStore:"""Manages PrefixCacheTracker instances across sessions.Keyed by session ID (from x-headroom-session-id header or computed hash).+ Within one session id, ``resolve_tracker`` keys trackers by conversation+ lineage so concurrent conversations sharing a fallback id do not thrash+ one tracker's frozen-prefix state (#2085).Automatically cleans up expired sessions."""@@ -776,6 +807,15 @@ class SessionTrackerStore:self._default_config = default_config or PrefixFreezeConfig()self._last_cleanup: float = time.time()self._cleanup_interval: float = 60.0 # Cleanup every 60s+ # Conversation lineages per session id: tracker key -> snapshot of the+ # canonicalized messages of the last request that lineage served. The+ # first lineage lives under the bare session id (single-conversation+ # sessions behave exactly as before); later lineages get unique+ # "<session_id>\x00<n>" keys — NUL cannot appear in an HTTP header+ # value, so a synthetic key can never collide with a client-supplied+ # x-headroom-session-id.+ self._lineages: dict[str, OrderedDict[str, list[Any]]] = {}+ self._lineage_counter = itertools.count(1)def get_or_create(self, session_id: str, provider: str) -> PrefixCacheTracker:"""Get existing tracker or create a new one for this session."""@@ -795,6 +835,122 @@ class SessionTrackerStore:self._trackers[session_id] = trackerreturn tracker+ def resolve_tracker(+ self,+ session_id: str,+ provider: str,+ messages: list[dict[str, Any]] | None = None,+ ) -> PrefixCacheTracker:+ """Resolve the tracker for THIS conversation within a session id (#2085).++ Concurrent conversations often share a fallback session id — same+ model + system prompt covers a Claude Code session together with its+ parallel subagents, or several sessions in one workspace. On a shared+ tracker their interleaved histories cross-contaminate the+ frozen-prefix state: freeze never stabilizes and the provider prompt+ cache is re-written on nearly every call.++ Lineage resolution keys trackers by conversation content instead:+ reuse the tracker whose previous request messages are a prefix of the+ incoming history (client histories are append-only, so a+ conversation's next request always extends its previous one); start a+ fresh lineage when the history diverges or was rewritten (client-side+ compaction — the provider cache line is gone then anyway).+ Byte-identical histories (templated fan-outs before they diverge)+ intentionally share a tracker: their provider cache line is identical+ too, so sharing is harmless.++ The session id itself is never altered, so session-sticky state keyed+ on it elsewhere (beta headers, CCR/memory registries, the compression+ cache) is unaffected.++ Args:+ session_id: Session identity from :meth:`compute_session_id`.+ provider: Provider name for a newly created tracker.+ messages: The request's original client messages, as captured+ right after the INPUT_RECEIVED pipeline extension and before+ security-scan/hook/image-compression mutation — the same+ snapshot ``update_from_response`` records, so the chain+ compares like against like across turns. ``None``/empty+ (legacy callers, stub stores in tests) falls back to plain+ :meth:`get_or_create`.++ Returns:+ The ``PrefixCacheTracker`` for this conversation's lineage.+ """+ if not messages or not self._default_config.enabled:+ # No lineage signal, or prefix freeze is disabled (there is no+ # frozen state to protect): legacy one-tracker-per-session-id.+ return self.get_or_create(session_id, provider)++ # Prune expired trackers BEFORE matching, so a dead lineage cannot win+ # the match. This also arms the cleanup interval: the get_or_create+ # calls below cannot re-trigger a prune mid-function, so the family+ # read here stays attached through the stamp at the end.+ self._maybe_cleanup()++ # The repo's canonical cross-turn equivalence, shared with the+ # cache-stable delta path: a moved cache breakpoint, string<->block+ # content sugar, or per-turn transport annotations must not read as+ # a rewritten history. Object comparison plus one structural snapshot+ # per request (~1ms on a 2MB history — same order as the handler's+ # existing request deepcopy; no serialization or hashing).+ canon = _canonicalize_for_prefix_compare(messages)+ if not canon:+ # Degenerate: every message projected away (pure directive+ # content) — no lineage signal to match on.+ return self.get_or_create(session_id, provider)+ snap = _lineage_snapshot(canon)++ family = self._lineages.setdefault(session_id, OrderedDict())++ # Longest recorded chain that prefixes the incoming history wins.+ best_key: str | None = None+ best_len = -1+ for key, chain in family.items():+ if len(chain) > len(snap) or len(chain) <= best_len:+ continue+ if snap[: len(chain)] == chain:+ best_key, best_len = key, len(chain)++ if best_key is None:+ cap = self._default_config.max_lineages_per_session+ if len(family) >= cap:+ # Family is full: over-cap conversations share one overflow+ # tracker instead of evicting an established lineage. Any+ # eviction policy degrades EVERY conversation once the+ # working set exceeds the cap (under round-robin the victim+ # is always the conversation about to arrive), while overflow+ # sharing degrades only the over-cap tail — to exactly the+ # pre-lineage shared-tracker behavior — and established+ # lineages keep their state. A cap <= 0 therefore disables+ # lineage splitting entirely.+ overflow_key = f"{session_id}\x00overflow"+ if overflow_key not in self._trackers:+ logger.info(+ "SessionTrackerStore: lineage cap %d reached for session %s; "+ "over-cap conversations share an overflow tracker (raise "+ "PrefixFreezeConfig.max_lineages_per_session if this "+ "workspace genuinely runs more concurrent conversations)",+ cap,+ session_id,+ )+ return self.get_or_create(overflow_key, provider)+ if not family:+ # First lineage rides the bare session id; this also adopts a+ # tracker created earlier via plain get_or_create.+ best_key = session_id+ else:+ best_key = f"{session_id}\x00{next(self._lineage_counter)}"++ # get_or_create (not a private fetch) so test stubs that patch the+ # instance method keep intercepting tracker creation; its internal+ # cleanup is interval-gated and was armed above, so it cannot prune+ # the family before the stamp below.+ tracker = self.get_or_create(best_key, provider)+ family[best_key] = snap+ return tracker+def compute_session_id(self,request: Any,@@ -807,13 +963,20 @@ class SessionTrackerStore:1. x-headroom-session-id header (explicit)2. Hash of (model + system prompt) — stable per conversation- The system prompt is harvested from ``role:"system"`` entries in- ``messages``. Anthropic carries the system prompt as a top-level- ``body["system"]`` field instead, so its handler prepends that as a- synthetic ``role:"system"`` message before calling this — otherwise every+ The system prompt is harvested from the LEADING run of ``role:"system"``+ entries in ``messages`` (everything before the first non-system turn).+ Anthropic carries the system prompt as a top-level ``body["system"]``+ field instead, so its handler prepends that as a synthetic+ ``role:"system"`` message before calling this — otherwise everyAnthropic conversation on the same model would collapse to one session idand their session-sticky state (CCR/memory tools, beta headers, frozenprefix) would cross-contaminate.++ Only the leading run counts: agentic clients (Claude Code) interleave+ ``role:"system"`` reminder turns INTO the history as it grows (hook+ output, skills lists, truncation notices). Hashing those would rotate+ the session id mid-conversation, orphaning the prefix tracker and every+ other session-sticky subsystem each time a reminder lands (#2085)."""# Check for explicit session headerif hasattr(request, "headers"):@@ -821,17 +984,18 @@ class SessionTrackerStore:if session_header:return str(session_header)- # Fall back to hashing model + all system-text content.+ # Fall back to hashing model + the leading system-text run.system_parts: list[str] = []for msg in messages:- if msg.get("role") == "system":- content = msg.get("content", "")- if isinstance(content, str):- system_parts.append(content)- elif isinstance(content, list):- for block in content:- if isinstance(block, dict) and block.get("type") == "text":- system_parts.append(block.get("text", ""))+ if msg.get("role") != "system":+ break+ content = msg.get("content", "")+ if isinstance(content, str):+ system_parts.append(content)+ elif isinstance(content, list):+ for block in content:+ if isinstance(block, dict) and block.get("type") == "text":+ system_parts.append(block.get("text", ""))system_content = json.dumps(system_parts, ensure_ascii=False, separators=(",", ":"))key = f"{model}:{system_content}"@@ -848,11 +1012,18 @@ class SessionTrackerStore:del self._trackers[sid]if expired:+ # Keep the lineage index in step with the tracker map.+ for base in list(self._lineages):+ family = self._lineages[base]+ for key in [k for k in family if k not in self._trackers]:+ del family[key]+ if not family:+ del self._lineages[base]logger.debug("SessionTrackerStore: cleaned up %d expired sessions", len(expired))self._last_cleanup = now@propertydef active_sessions(self) -> int:- """Number of active session trackers."""+ """Number of active session trackers (one per conversation lineage)."""return len(self._trackers)diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.pyindex c3c3e0547..427ee6413 100644--- a/headroom/proxy/handlers/anthropic.py+++ b/headroom/proxy/handlers/anthropic.py@@ -1069,16 +1069,32 @@ class AnthropicHandlerMixin:# session id (and its sticky CCR/memory tools, beta headers, and# frozen-prefix state). This synthetic message only derives the id; it# is never forwarded.+ # Both the session id and the conversation lineage derive from the+ # SAME original client bytes (pre security-scan/hook mutation,+ # which may rewrite messages differently turn-to-turn — a+ # turn-dependent rewrite of the leading system text would rotate+ # the id mid-conversation, orphaning all session-sticky state).+ # The synthetic system message folds the top-level system prompt+ # into both derivations, so conversations sharing an explicit+ # header id but differing in system prompt keep separate lineages.system_prompt = body.get("system")session_messages = (- [{"role": "system", "content": system_prompt}, *messages]+ [{"role": "system", "content": system_prompt}, *original_client_messages]if system_prompt is not None- else messages+ else original_client_messages)session_id = self.session_tracker_store.compute_session_id(request, model, session_messages)- prefix_tracker = self.session_tracker_store.get_or_create(session_id, "anthropic")+ # Resolve the tracker by conversation lineage within the session id+ # (#2085): one model + system prompt spans a Claude Code session and+ # all its parallel subagents, so concurrent conversations share this+ # fallback id — on one shared tracker their interleaved histories+ # thrash the frozen-prefix state and the provider prompt cache is+ # re-written on nearly every call.+ prefix_tracker = self.session_tracker_store.resolve_tracker(+ session_id, "anthropic", messages=session_messages+ )frozen_message_count = prefix_tracker.get_frozen_message_count()# Idle gap since the previous turn's response, snapshotted at fetch# (before get_or_create bumped the access clock). Forwarded to thediff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.pyindex 69ee8c369..86e25fa1e 100644--- a/headroom/proxy/handlers/openai.py+++ b/headroom/proxy/handlers/openai.py@@ -2683,10 +2683,19 @@ class OpenAIHandlerMixin:optimized_messages = messagesoptimized_tokens = original_tokens- # Get prefix cache tracker for this session- openai_session_id = self.session_tracker_store.compute_session_id(request, model, messages)- openai_prefix_tracker = self.session_tracker_store.get_or_create(- openai_session_id, "openai"+ # Get prefix cache tracker for this session, resolved by conversation+ # lineage within the session id (#2085) so concurrent conversations+ # sharing a fallback id (same model + system prompt) do not thrash one+ # tracker's frozen-prefix state. Both the id and the lineage derive+ # from the SAME original client bytes — pre image compression and+ # pre_compress hook, both of which may rewrite history bytes+ # differently turn-to-turn (a turn-dependent rewrite of the leading+ # system text would rotate the id mid-conversation).+ openai_session_id = self.session_tracker_store.compute_session_id(+ request, model, original_client_messages+ )+ openai_prefix_tracker = self.session_tracker_store.resolve_tracker(+ openai_session_id, "openai", messages=original_client_messages)# PR-A6 (P5-50, preps P0-6): session-sticky `OpenAI-Beta` merge.@@ -5292,7 +5301,9 @@ class OpenAIHandlerMixin:upstream: Any = Nonefrom headroom.proxy.helpers import merge_extra_headers- upstream_headers = merge_extra_headers(upstream_headers, self.config.openai_extra_headers)+ upstream_headers = merge_extra_headers(+ upstream_headers, self.config.openai_extra_headers+ )for ws_attempt in range(ws_connect_attempts):try:diff --git a/headroom/proxy/handlers/streaming.py b/headroom/proxy/handlers/streaming.pyindex d4007e141..53a241e93 100644--- a/headroom/proxy/handlers/streaming.py+++ b/headroom/proxy/handlers/streaming.py@@ -70,7 +70,12 @@ class StreamingMixin:def _get_session_key(body: dict, session_header: str | None = None) -> str:"""Return session identity from an explicit header or a body-derived hash.- Fallback mirrors prefix_tracker.compute_session_id: md5(model:system[:500]).+ The fallback is a coarse, self-contained hash (md5 of+ ``model:system[:500]``) used ONLY to key the mid-turn steering state+ below. It is NOT the same derivation as+ ``prefix_tracker.compute_session_id`` (which json-encodes the full+ leading system-text run) — do not key cross-subsystem state on the+ assumption that the two fallbacks agree."""if session_header:return session_headerdiff --git a/tests/test_anthropic_pre_upstream_backpressure.py b/tests/test_anthropic_pre_upstream_backpressure.pyindex 805bb9ec0..3595fc429 100644--- a/tests/test_anthropic_pre_upstream_backpressure.py+++ b/tests/test_anthropic_pre_upstream_backpressure.py@@ -172,6 +172,14 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):update_from_response=lambda *a, **k: None,record_request=lambda *a, **k: None,),+ resolve_tracker=lambda *a, **k: SimpleNamespace(+ _cached_token_count=0,+ get_frozen_message_count=lambda: 0,+ get_last_original_messages=lambda: [],+ get_last_forwarded_messages=lambda: [],+ update_from_response=lambda *a, **k: None,+ record_request=lambda *a, **k: None,+ ),)# Unit 4: the only field this test cares about.self.anthropic_pre_upstream_sem = anthropic_pre_upstream_semdiff --git a/tests/test_anthropic_stage_timings.py b/tests/test_anthropic_stage_timings.pyindex cd4177310..93b3e1944 100644--- a/tests/test_anthropic_stage_timings.py+++ b/tests/test_anthropic_stage_timings.py@@ -102,6 +102,12 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):get_last_forwarded_messages=lambda: [],record_request=lambda *a, **k: None,),+ resolve_tracker=lambda *a, **k: SimpleNamespace(+ get_frozen_message_count=lambda: 0,+ get_last_original_messages=lambda: [],+ get_last_forwarded_messages=lambda: [],+ record_request=lambda *a, **k: None,+ ),)async def _next_request_id(self) -> str:diff --git a/tests/test_cache/test_prefix_tracker.py b/tests/test_cache/test_prefix_tracker.pyindex 76af68bdd..69f39e59e 100644--- a/tests/test_cache/test_prefix_tracker.py+++ b/tests/test_cache/test_prefix_tracker.py@@ -370,8 +370,8 @@ class TestSessionTrackerStore:id3 = store.compute_session_id(MockRequest(), "gpt-4", messages)assert id3 != id1- def test_compute_session_id_uses_all_system_messages(self, store):- """Different dynamic system messages should not collide."""+ def test_compute_session_id_distinguishes_leading_system_run(self, store):+ """Different dynamic LEADING system messages should not collide."""class MockRequest:headers = {}@@ -454,6 +454,340 @@ class TestSessionTrackerStore:assert isinstance(session_id, str)assert len(session_id) == 16+ def test_mid_conversation_system_turns_do_not_rotate_session_id(self, store):+ """Claude Code sends <system-reminder> turns as role:"system" MESSAGES+ interleaved into the history (hook outputs, skills lists, truncation+ notices). Hashing those into the fallback id rotates the session id+ mid-conversation — orphaning the prefix tracker and every other+ session-sticky subsystem (beta headers, CCR/memory registries, the+ compression cache) each time a reminder lands. Only the LEADING run of+ system messages is session identity; later system turns are content.+ """++ class MockRequest:+ headers = {}++ leading = {"role": "system", "content": "You are an agent. " * 40}+ turn1 = [leading, {"role": "user", "content": "read file A"}]+ turn2 = turn1 + [+ {"role": "assistant", "content": "read it"},+ {"role": "user", "content": "tool result ..."},+ {+ "role": "system",+ "content": "<system-reminder>Truncated: PARTIAL view</system-reminder>",+ },+ {"role": "user", "content": "continue"},+ ]++ id1 = store.compute_session_id(MockRequest(), "claude-sonnet-5", turn1)+ id2 = store.compute_session_id(MockRequest(), "claude-sonnet-5", turn2)+ assert id1 == id2+++class TestConversationLineageResolution:+ """resolve_tracker: one PrefixCacheTracker per conversation lineage (#2085).++ Concurrent conversations that share a fallback session id (same model ++ same system prompt — e.g. a Claude Code session and its parallel subagents)+ must not thrash one tracker's frozen-prefix state: interleaved turns would+ each see the *other* conversation's prefix, freeze never stabilizes, and+ the provider prompt cache is re-written on every call.++ resolve_tracker keys trackers by message lineage instead: an incoming+ history that extends a known lineage reuses its tracker; a diverging or+ rewritten history gets a fresh one. The session id itself is never changed,+ so session-sticky state keyed on it elsewhere (beta headers, CCR/memory+ registries, the compression cache) is unaffected.+ """++ @pytest.fixture+ def store(self):+ return SessionTrackerStore()++ @staticmethod+ def _history(name: str, turn: int) -> list[dict]:+ """Client-shaped request messages for `turn` (1-based): u0,a0,...,u_{turn-1}."""+ messages: list[dict] = []+ for t in range(turn):+ messages.append({"role": "user", "content": f"[{name}] user {t} " + "x" * 200})+ if t < turn - 1:+ messages.append(+ {"role": "assistant", "content": f"[{name}] assistant {t} " + "y" * 200}+ )+ return messages++ @staticmethod+ def _block_history(turn: int, cc_on: int | None) -> list[dict]:+ """Block-content history; cache_control breakpoint on message `cc_on` (or none)."""+ messages: list[dict] = []+ for t in range(turn):+ messages.append(+ {"role": "user", "content": [{"type": "text", "text": f"user {t} " + "x" * 200}]}+ )+ if t < turn - 1:+ messages.append(+ {+ "role": "assistant",+ "content": [{"type": "text", "text": f"assistant {t} " + "y" * 200}],+ }+ )+ if cc_on is not None:+ msg = messages[cc_on]+ blocks = [dict(b) for b in msg["content"]]+ blocks[-1] = {**blocks[-1], "cache_control": {"type": "ephemeral"}}+ messages[cc_on] = {**msg, "content": blocks}+ return messages++ def test_interleaved_conversations_resolve_to_independent_trackers(self, store):+ """The #2085 production shape: two conversations, one session id,+ alternating requests. Each must keep its own tracker and per-turn state+ (on a shared tracker, _turn_number would count both conversations)."""+ sid = "shared-fallback-id"+ trackers: dict[str, PrefixCacheTracker] = {}+ for turn in range(1, 5):+ for name in ("A", "B"):+ history = self._history(name, turn)+ tracker = store.resolve_tracker(sid, "anthropic", messages=history)+ trackers.setdefault(name, tracker)+ assert tracker is trackers[name], f"[{name}] turn {turn} switched trackers"+ tracker.update_from_response(+ cache_read_tokens=1000 * turn,+ cache_write_tokens=500,+ messages=history,+ )+ assert trackers["A"] is not trackers["B"]+ assert trackers["A"]._turn_number == 4+ assert trackers["B"]._turn_number == 4++ def test_identical_first_turns_share_until_divergence_then_split(self, store):+ """Templated fan-outs send byte-identical first turns. While histories+ are identical, sharing a tracker is harmless (the provider cache line+ is identical too); they must split as soon as the histories diverge."""+ sid = "shared"+ first = [{"role": "user", "content": "verify the fix " + "p" * 300}]+ t_a1 = store.resolve_tracker(sid, "anthropic", messages=first)+ t_b1 = store.resolve_tracker(sid, "anthropic", messages=first)+ assert t_b1 is t_a1++ a2 = first + [+ {"role": "assistant", "content": "answer A"},+ {"role": "user", "content": "next A"},+ ]+ b2 = first + [+ {"role": "assistant", "content": "answer B"},+ {"role": "user", "content": "next B"},+ ]+ t_a2 = store.resolve_tracker(sid, "anthropic", messages=a2)+ t_b2 = store.resolve_tracker(sid, "anthropic", messages=b2)+ assert t_a2 is not t_b2++ a3 = a2 + [+ {"role": "assistant", "content": "answer A2"},+ {"role": "user", "content": "next A2"},+ ]+ assert store.resolve_tracker(sid, "anthropic", messages=a3) is t_a2++ @pytest.mark.parametrize(+ "cc_turn2",+ [0, -1, None],+ ids=["breakpoint-stays", "breakpoint-moved-to-last", "breakpoint-removed"],+ )+ def test_cache_control_movement_does_not_split_lineage(self, store, cc_turn2):+ """Clients move the cache_control breakpoint every turn; that must not+ read as a rewritten history."""+ sid = "shared"+ t1 = store.resolve_tracker(sid, "anthropic", messages=self._block_history(1, cc_on=0))+ t2 = store.resolve_tracker(+ sid, "anthropic", messages=self._block_history(2, cc_on=cc_turn2)+ )+ assert t2 is t1++ @pytest.mark.parametrize(+ "requote",+ [+ lambda m: {**m, "content": [{"type": "text", "text": m["content"]}]},+ lambda m: {+ **m,+ "content": [{"type": "text", "text": m["content"], "index": 0}],+ },+ lambda m: {+ **m,+ "content": [+ {"type": "text", "text": m["content"]},+ {"cachePoint": {"type": "default"}},+ ],+ },+ ],+ ids=["string-to-block-sugar", "streaming-index-annotation", "bedrock-cachepoint-block"],+ )+ def test_representation_churn_does_not_split_lineage(self, store, requote):+ """Clients re-encode history turn-to-turn without changing content+ (litellm flips string<->block sugar, streaming assembly adds `index`,+ Bedrock moves its cachePoint block). Lineage matching must use the+ same canonical equivalence as the cache-stable delta path."""+ sid = "shared"+ first = {"role": "user", "content": "hello " + "x" * 200}+ t1 = store.resolve_tracker(sid, "anthropic", messages=[first])+ grown = [+ requote(first),+ {"role": "assistant", "content": "hi"},+ {"role": "user", "content": "next"},+ ]+ assert store.resolve_tracker(sid, "anthropic", messages=grown) is t1++ @pytest.mark.parametrize(+ "rewrite",+ [+ lambda h: [{"role": "user", "content": "[summary of the conversation so far]"}],+ lambda h: [h[0], {"role": "assistant", "content": "EDITED"}, *h[2:]],+ lambda h: h[:-2],+ ],+ ids=["compacted", "middle-edited", "truncated"],+ )+ def test_rewritten_history_gets_fresh_tracker(self, store, rewrite):+ """A rewritten history (client-side /compact, edits, truncation) means+ the provider cache line is gone anyway: start a fresh lineage, keep the+ old tracker until TTL, and never touch the session id."""+ sid = "shared"+ history = self._history("A", 3)+ tracker = store.resolve_tracker(sid, "anthropic", messages=history)+ fresh = store.resolve_tracker(sid, "anthropic", messages=rewrite(history))+ assert fresh is not tracker+ assert store.active_sessions == 2++ @pytest.mark.parametrize("messages", [None, []], ids=["none", "empty"])+ def test_resolve_without_messages_matches_legacy_get_or_create(self, store, messages):+ tracker = store.resolve_tracker("sid", "anthropic", messages=messages)+ assert tracker is store.get_or_create("sid", "anthropic")++ def test_resolve_with_freeze_disabled_matches_legacy_get_or_create(self):+ """With prefix freeze off there is no frozen state to protect — skip+ lineage bookkeeping entirely."""+ store = SessionTrackerStore(PrefixFreezeConfig(enabled=False))+ t_a = store.resolve_tracker("sid", "anthropic", messages=self._history("A", 1))+ t_b = store.resolve_tracker("sid", "anthropic", messages=self._history("B", 1))+ assert t_a is t_b+ assert t_a is store.get_or_create("sid", "anthropic")++ def test_over_cap_conversations_share_one_overflow_tracker(self):+ """A fan-out storm on one session id must not grow trackers unbounded:+ past the cap, new conversations share one overflow tracker instead of+ evicting an established lineage."""+ store = SessionTrackerStore(PrefixFreezeConfig(max_lineages_per_session=4))+ sid = "storm"+ overflow = set()+ for i in range(9):+ tracker = store.resolve_tracker(+ sid, "anthropic", messages=[{"role": "user", "content": f"task {i} " + "z" * 100}]+ )+ if i >= 4:+ overflow.add(id(tracker))+ assert store.active_sessions == 5 # 4 lineages + 1 shared overflow+ assert len(overflow) == 1++ def test_established_lineages_survive_cap_overflow(self):+ """Filling the family must never evict an established conversation —+ under round-robin any eviction victim is the next requester, which+ would degrade EVERY conversation to a cold tracker per turn."""+ store = SessionTrackerStore(PrefixFreezeConfig(max_lineages_per_session=2))+ sid = "shared"+ parent_history = self._history("parent", 4)+ parent = store.resolve_tracker(sid, "anthropic", messages=parent_history)+ shorty = store.resolve_tracker(sid, "anthropic", messages=self._history("shorty", 1))+ # Third divergent conversation lands on the overflow tracker; both+ # established lineages keep their trackers.+ newcomer = store.resolve_tracker(sid, "anthropic", messages=self._history("new", 1))+ assert newcomer is not parent+ assert newcomer is not shorty+ grown = parent_history + [+ {"role": "assistant", "content": "[parent] assistant 3 " + "y" * 200},+ {"role": "user", "content": "[parent] user 4 " + "x" * 200},+ ]+ assert store.resolve_tracker(sid, "anthropic", messages=grown) is parent+ assert (+ store.resolve_tracker(sid, "anthropic", messages=self._history("shorty", 2)) is shorty+ )++ def test_no_cliff_at_cap_plus_one_round_robin(self):+ """cap+1 conversations round-robining turns: the in-cap conversations+ keep their trackers on every round (no eviction churn); only the+ over-cap tail shares the overflow tracker."""+ store = SessionTrackerStore(PrefixFreezeConfig(max_lineages_per_session=3))+ sid = "shared"+ trackers: dict[str, PrefixCacheTracker] = {}+ for turn in range(1, 4):+ for name in ("A", "B", "C", "D"):+ tracker = store.resolve_tracker(+ sid, "anthropic", messages=self._history(name, turn)+ )+ if turn == 1:+ trackers[name] = tracker+ elif name != "D":+ assert tracker is trackers[name], f"[{name}] turn {turn} lost its tracker"+ else:+ assert tracker is trackers["D"] # stable overflow tracker++ def test_empty_canonical_history_falls_back_to_legacy(self):+ """A history whose every message projects away (pure directive+ content) carries no lineage signal — behave like get_or_create."""+ store = SessionTrackerStore()+ tracker = store.resolve_tracker("sid", "anthropic", messages=[{}])+ assert tracker is store.get_or_create("sid", "anthropic")++ def test_nan_in_tool_payload_does_not_split_lineage(self):+ """json.loads accepts bare NaN, and NaN != NaN — a resent history+ containing one must still read as the same conversation."""+ store = SessionTrackerStore()+ sid = "shared"+ turn1 = [+ {"role": "user", "content": "run the tool " + "x" * 200},+ {+ "role": "assistant",+ "content": [+ {"type": "tool_use", "id": "t1", "name": "score", "input": {"v": float("nan")}}+ ],+ },+ {+ "role": "user",+ "content": [{"type": "tool_result", "tool_use_id": "t1", "content": "ok"}],+ },+ ]+ t1 = store.resolve_tracker(sid, "anthropic", messages=turn1)+ turn2 = turn1 + [+ {"role": "assistant", "content": "done"},+ {"role": "user", "content": "next"},+ ]+ assert store.resolve_tracker(sid, "anthropic", messages=turn2) is t1++ def test_expired_lineages_are_cleaned_up(self):+ config = PrefixFreezeConfig(session_ttl_seconds=1)+ store = SessionTrackerStore(default_config=config)+ for name in ("A", "B"):+ tracker = store.resolve_tracker("sid", "anthropic", messages=self._history(name, 1))+ tracker._last_activity = time.time() - 2++ store._last_cleanup = 0+ store._maybe_cleanup()+ assert store.active_sessions == 0++ # The lineage index must not resurrect evicted trackers: extending an+ # evicted conversation starts cold.+ fresh = store.resolve_tracker("sid", "anthropic", messages=self._history("A", 2))+ assert fresh._turn_number == 0++ def test_shared_session_id_is_not_rotated(self, store):+ """Composition guard: lineage resolution must not leak into session-id+ derivation — beta stickiness and the compression cache key on it."""++ class MockRequest:+ headers = {}++ msgs_a = [{"role": "system", "content": "S"}, *self._history("A", 1)]+ msgs_b = [{"role": "system", "content": "S"}, *self._history("B", 1)]+ id_a = store.compute_session_id(MockRequest(), "claude-3", msgs_a)+ id_b = store.compute_session_id(MockRequest(), "claude-3", msgs_b)+ assert id_a == id_b+class TestMultiTurnScenario:"""Integration-style tests simulating multi-turn conversations."""diff --git a/tests/test_cold_start_fast_pass.py b/tests/test_cold_start_fast_pass.pyindex 6c6baa4c6..a230facb4 100644--- a/tests/test_cold_start_fast_pass.py+++ b/tests/test_cold_start_fast_pass.py@@ -139,6 +139,12 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):get_last_forwarded_messages=lambda: [],record_request=lambda *a, **k: None,),+ resolve_tracker=lambda *a, **k: SimpleNamespace(+ get_frozen_message_count=lambda: 0,+ get_last_original_messages=lambda: [],+ get_last_forwarded_messages=lambda: [],+ record_request=lambda *a, **k: None,+ ),)# Cold-start deferral wiring under test.self._background_compression_enabled = True