Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(cache): stable session identity and per-conversation prefix trackers under agentic clients (#2193)

Andrei Boldyrev7bfb1d7f38ac
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 388573cbd..63239d167 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **install:** `headroom install apply --env KEY=VALUE` (repeatable) passes environment variables into supervised runners (macOS launchd, Linux systemd/cron, Windows services/tasks). These runners previously started with a bare environment and did not inherit the interactive shell's exports — e.g. a custom `HEADROOM_WORKSPACE_DIR` never reached the supervised process, so `headroom install agent run` looked for its manifest in the wrong location and failed outright even though `install apply` itself succeeded. `--env` values are merged into `DeploymentManifest.base_env` last, so they can override auto-derived defaults, and are threaded into the generated `run-headroom.sh`/`ensure-headroom.sh` (and Windows equivalents) as `export`/`$env:` lines before the `exec`.
### Fixed
+- **cache/prefix-freeze:** resolve `PrefixCacheTracker`s per conversation lineage within a session id, so concurrent conversations sharing a fallback id no longer thrash one tracker's frozen-prefix state ([#2085](https://github.com/headroomlabs-ai/headroom/issues/2085)). Without an `x-headroom-session-id` header the fallback id hashes `model + system prompt` — identical across a Claude Code session and every one of its parallel subagents (and any set of sessions reusing one system prompt). On the shared tracker their interleaved histories cross-contaminate the freeze state: the forwarded prefix is byte-unstable on nearly every turn and the provider prompt cache is re-written instead of read — reported as ~4.4x cache-creation inflation and a 2.5–3x net cost increase under Claude Code. `SessionTrackerStore.resolve_tracker` now reuses the tracker whose previous request messages are a prefix of the incoming history (client histories are append-only, so a conversation's next request always extends its previous one), starts a fresh lineage when the history diverges or was rewritten (client-side compaction — that provider cache line is gone anyway), and caps lineages per session id (`PrefixFreezeConfig.max_lineages_per_session`, default 32; over-cap conversations share one overflow tracker instead of evicting established lineages, so a fan-out storm past the cap degrades only its own tail — and `0` disables lineage splitting). Matching compares the original client bytes under the same canonical cross-turn equivalence as the cache-stable delta path (`_canonicalize_for_prefix_compare`), so a moved cache breakpoint, string<->block content sugar, or per-turn transport annotations do not read as a rewrite. Separately, the fallback id now hashes only the LEADING run of `role:"system"` messages: agentic clients interleave `<system-reminder>` turns into the history as actual system-role messages (hook output, skills lists, truncation notices), and hashing those rotated the session id mid-conversation — orphaning the prefix tracker and every other session-sticky subsystem (beta headers, CCR/memory registries, the compression cache) each time a reminder landed. Both handler paths now derive the session id and the lineage from the same original client bytes, so a turn-dependent hook rewrite cannot rotate one without the other. The session id itself never changes: session-sticky state keyed on it (beta-header stickiness, CCR and memory-tool registries, the compression cache) is untouched, and a single-conversation session keeps its exact previous behavior (the first lineage lives under the bare id).
- **proxy/bedrock:** wire `PrefixCacheTracker` updates into both Bedrock backend paths (`handle_anthropic_messages`'s non-streaming branch in `anthropic.py`, and `_stream_response_bedrock` in `streaming.py`). `update_from_response()` was previously only called from the direct-Anthropic-API branch; both Bedrock branches returned before ever reaching it, so the tracker's state stayed permanently empty for the life of a session on any `--backend bedrock` deployment: `extract_cache_stable_delta()` always saw no previous turn, and `--mode cache` fell back to full unmodified passthrough on every turn instead of freezing the already-cached prefix and compressing only the new suffix.
- **install:** `install_supervisor`'s macOS branch did an unconditional `launchctl bootout` followed by a bare `bootstrap` with no retry, unlike `start_supervisor`, which already rides out the ~15s EIO (error 5) window launchd exhibits for several seconds after a bootout. This left `install apply`'s own reinstall path (and anything that re-applies a deployment, e.g. a future `headroom doctor --fix`) exposed to a race that previously required manual recovery (bootout + remove the plist + reapply). Extracted the retry loop already used by `start_supervisor` into a shared `_bootstrap_with_retry()` helper, now used by both call sites.
- **proxy/savings:** `SavingsTracker.record_request()` only appended a history point when `tokens_saved > 0` (headroom's own lossy compression). In `--mode cache`, `tokens_saved` is near-always 0 by design, since the frozen prefix is byte-replayed rather than compressed to keep the provider's prompt cache warm. That silently dropped every history point on a cache-mode deployment even when `cache_read_tokens`/`cache_savings_usd` were large, making `headroom-monthly`-style tooling read as a total savings collapse. The guard now fires on `tokens_saved` OR `cache_read_tokens`, and the appended entry carries `cache_read_tokens`/`cache_savings_usd` so downstream consumers can show them; `_normalize_history_entry` defaults both fields to 0/0.0 for legacy entries that predate this change.
diff --git a/headroom/cache/prefix_tracker.py b/headroom/cache/prefix_tracker.py
index 585427447..d7e62f436 100644
--- a/headroom/cache/prefix_tracker.py
+++ b/headroom/cache/prefix_tracker.py
@@ -18,9 +18,11 @@ from __future__ import annotations
import copy
import hashlib
+import itertools
import json
import logging
import time
+from collections import OrderedDict
from dataclasses import dataclass
from typing import Any
@@ -70,6 +72,12 @@ class PrefixFreezeConfig:
# default in `_PROVIDER_CACHE_TTL_SECONDS`. Set to 3600 for a session that
# uses Anthropic's 1h cache breakpoint so idle-gap attribution stays honest.
cache_ttl_seconds: int | None = None
+ # Cap on concurrent conversation lineages tracked per session id (#2085).
+ # A fan-out storm (many parallel subagents sharing one fallback id) evicts
+ # the shortest-chain lineage instead of growing without bound. Raise it
+ # for workspaces that genuinely run more concurrent conversations on one
+ # model + system prompt.
+ max_lineages_per_session: int = 32
@dataclass
@@ -764,10 +772,33 @@ class PrefixCacheTracker:
return counts
+def _lineage_snapshot(obj: Any) -> Any:
+ """Structural copy of a canonical projection for lineage-chain storage.
+
+ Copies dict/list structure (immutable leaves are shared, so this costs
+ structure, not bytes) to isolate the stored chain from downstream in-place
+ mutation of the request, and normalizes NaN to a sentinel — ``json.loads``
+ accepts bare ``NaN`` and ``NaN != NaN``, so a byte-identical resend would
+ otherwise read as a rewritten history on every turn. Values inside opaque
+ tool payloads are walked for copying only; no keys are dropped (see
+ ``_OPAQUE_PAYLOAD_KEYS``).
+ """
+ if isinstance(obj, dict):
+ return {k: _lineage_snapshot(v) for k, v in obj.items()}
+ if isinstance(obj, list):
+ return [_lineage_snapshot(v) for v in obj]
+ if isinstance(obj, float) and obj != obj:
+ return "\x00nan"
+ return obj
+
+
class SessionTrackerStore:
"""Manages PrefixCacheTracker instances across sessions.
Keyed by session ID (from x-headroom-session-id header or computed hash).
+ Within one session id, ``resolve_tracker`` keys trackers by conversation
+ lineage so concurrent conversations sharing a fallback id do not thrash
+ one tracker's frozen-prefix state (#2085).
Automatically cleans up expired sessions.
"""
@@ -776,6 +807,15 @@ class SessionTrackerStore:
self._default_config = default_config or PrefixFreezeConfig()
self._last_cleanup: float = time.time()
self._cleanup_interval: float = 60.0 # Cleanup every 60s
+ # Conversation lineages per session id: tracker key -> snapshot of the
+ # canonicalized messages of the last request that lineage served. The
+ # first lineage lives under the bare session id (single-conversation
+ # sessions behave exactly as before); later lineages get unique
+ # "<session_id>\x00<n>" keys — NUL cannot appear in an HTTP header
+ # value, so a synthetic key can never collide with a client-supplied
+ # x-headroom-session-id.
+ self._lineages: dict[str, OrderedDict[str, list[Any]]] = {}
+ self._lineage_counter = itertools.count(1)
def get_or_create(self, session_id: str, provider: str) -> PrefixCacheTracker:
"""Get existing tracker or create a new one for this session."""
@@ -795,6 +835,122 @@ class SessionTrackerStore:
self._trackers[session_id] = tracker
return tracker
+ def resolve_tracker(
+ self,
+ session_id: str,
+ provider: str,
+ messages: list[dict[str, Any]] | None = None,
+ ) -> PrefixCacheTracker:
+ """Resolve the tracker for THIS conversation within a session id (#2085).
+
+ Concurrent conversations often share a fallback session id — same
+ model + system prompt covers a Claude Code session together with its
+ parallel subagents, or several sessions in one workspace. On a shared
+ tracker their interleaved histories cross-contaminate the
+ frozen-prefix state: freeze never stabilizes and the provider prompt
+ cache is re-written on nearly every call.
+
+ Lineage resolution keys trackers by conversation content instead:
+ reuse the tracker whose previous request messages are a prefix of the
+ incoming history (client histories are append-only, so a
+ conversation's next request always extends its previous one); start a
+ fresh lineage when the history diverges or was rewritten (client-side
+ compaction — the provider cache line is gone then anyway).
+ Byte-identical histories (templated fan-outs before they diverge)
+ intentionally share a tracker: their provider cache line is identical
+ too, so sharing is harmless.
+
+ The session id itself is never altered, so session-sticky state keyed
+ on it elsewhere (beta headers, CCR/memory registries, the compression
+ cache) is unaffected.
+
+ Args:
+ session_id: Session identity from :meth:`compute_session_id`.
+ provider: Provider name for a newly created tracker.
+ messages: The request's original client messages, as captured
+ right after the INPUT_RECEIVED pipeline extension and before
+ security-scan/hook/image-compression mutation — the same
+ snapshot ``update_from_response`` records, so the chain
+ compares like against like across turns. ``None``/empty
+ (legacy callers, stub stores in tests) falls back to plain
+ :meth:`get_or_create`.
+
+ Returns:
+ The ``PrefixCacheTracker`` for this conversation's lineage.
+ """
+ if not messages or not self._default_config.enabled:
+ # No lineage signal, or prefix freeze is disabled (there is no
+ # frozen state to protect): legacy one-tracker-per-session-id.
+ return self.get_or_create(session_id, provider)
+
+ # Prune expired trackers BEFORE matching, so a dead lineage cannot win
+ # the match. This also arms the cleanup interval: the get_or_create
+ # calls below cannot re-trigger a prune mid-function, so the family
+ # read here stays attached through the stamp at the end.
+ self._maybe_cleanup()
+
+ # The repo's canonical cross-turn equivalence, shared with the
+ # cache-stable delta path: a moved cache breakpoint, string<->block
+ # content sugar, or per-turn transport annotations must not read as
+ # a rewritten history. Object comparison plus one structural snapshot
+ # per request (~1ms on a 2MB history — same order as the handler's
+ # existing request deepcopy; no serialization or hashing).
+ canon = _canonicalize_for_prefix_compare(messages)
+ if not canon:
+ # Degenerate: every message projected away (pure directive
+ # content) — no lineage signal to match on.
+ return self.get_or_create(session_id, provider)
+ snap = _lineage_snapshot(canon)
+
+ family = self._lineages.setdefault(session_id, OrderedDict())
+
+ # Longest recorded chain that prefixes the incoming history wins.
+ best_key: str | None = None
+ best_len = -1
+ for key, chain in family.items():
+ if len(chain) > len(snap) or len(chain) <= best_len:
+ continue
+ if snap[: len(chain)] == chain:
+ best_key, best_len = key, len(chain)
+
+ if best_key is None:
+ cap = self._default_config.max_lineages_per_session
+ if len(family) >= cap:
+ # Family is full: over-cap conversations share one overflow
+ # tracker instead of evicting an established lineage. Any
+ # eviction policy degrades EVERY conversation once the
+ # working set exceeds the cap (under round-robin the victim
+ # is always the conversation about to arrive), while overflow
+ # sharing degrades only the over-cap tail — to exactly the
+ # pre-lineage shared-tracker behavior — and established
+ # lineages keep their state. A cap <= 0 therefore disables
+ # lineage splitting entirely.
+ overflow_key = f"{session_id}\x00overflow"
+ if overflow_key not in self._trackers:
+ logger.info(
+ "SessionTrackerStore: lineage cap %d reached for session %s; "
+ "over-cap conversations share an overflow tracker (raise "
+ "PrefixFreezeConfig.max_lineages_per_session if this "
+ "workspace genuinely runs more concurrent conversations)",
+ cap,
+ session_id,
+ )
+ return self.get_or_create(overflow_key, provider)
+ if not family:
+ # First lineage rides the bare session id; this also adopts a
+ # tracker created earlier via plain get_or_create.
+ best_key = session_id
+ else:
+ best_key = f"{session_id}\x00{next(self._lineage_counter)}"
+
+ # get_or_create (not a private fetch) so test stubs that patch the
+ # instance method keep intercepting tracker creation; its internal
+ # cleanup is interval-gated and was armed above, so it cannot prune
+ # the family before the stamp below.
+ tracker = self.get_or_create(best_key, provider)
+ family[best_key] = snap
+ return tracker
+
def compute_session_id(
self,
request: Any,
@@ -807,13 +963,20 @@ class SessionTrackerStore:
1. x-headroom-session-id header (explicit)
2. Hash of (model + system prompt) — stable per conversation
- The system prompt is harvested from ``role:"system"`` entries in
- ``messages``. Anthropic carries the system prompt as a top-level
- ``body["system"]`` field instead, so its handler prepends that as a
- synthetic ``role:"system"`` message before calling this — otherwise every
+ The system prompt is harvested from the LEADING run of ``role:"system"``
+ entries in ``messages`` (everything before the first non-system turn).
+ Anthropic carries the system prompt as a top-level ``body["system"]``
+ field instead, so its handler prepends that as a synthetic
+ ``role:"system"`` message before calling this — otherwise every
Anthropic conversation on the same model would collapse to one session id
and their session-sticky state (CCR/memory tools, beta headers, frozen
prefix) would cross-contaminate.
+
+ Only the leading run counts: agentic clients (Claude Code) interleave
+ ``role:"system"`` reminder turns INTO the history as it grows (hook
+ output, skills lists, truncation notices). Hashing those would rotate
+ the session id mid-conversation, orphaning the prefix tracker and every
+ other session-sticky subsystem each time a reminder lands (#2085).
"""
# Check for explicit session header
if hasattr(request, "headers"):
@@ -821,17 +984,18 @@ class SessionTrackerStore:
if session_header:
return str(session_header)
- # Fall back to hashing model + all system-text content.
+ # Fall back to hashing model + the leading system-text run.
system_parts: list[str] = []
for msg in messages:
- if msg.get("role") == "system":
- content = msg.get("content", "")
- if isinstance(content, str):
- system_parts.append(content)
- elif isinstance(content, list):
- for block in content:
- if isinstance(block, dict) and block.get("type") == "text":
- system_parts.append(block.get("text", ""))
+ if msg.get("role") != "system":
+ break
+ content = msg.get("content", "")
+ if isinstance(content, str):
+ system_parts.append(content)
+ elif isinstance(content, list):
+ for block in content:
+ if isinstance(block, dict) and block.get("type") == "text":
+ system_parts.append(block.get("text", ""))
system_content = json.dumps(system_parts, ensure_ascii=False, separators=(",", ":"))
key = f"{model}:{system_content}"
@@ -848,11 +1012,18 @@ class SessionTrackerStore:
del self._trackers[sid]
if expired:
+ # Keep the lineage index in step with the tracker map.
+ for base in list(self._lineages):
+ family = self._lineages[base]
+ for key in [k for k in family if k not in self._trackers]:
+ del family[key]
+ if not family:
+ del self._lineages[base]
logger.debug("SessionTrackerStore: cleaned up %d expired sessions", len(expired))
self._last_cleanup = now
@property
def active_sessions(self) -> int:
- """Number of active session trackers."""
+ """Number of active session trackers (one per conversation lineage)."""
return len(self._trackers)
diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.py
index c3c3e0547..427ee6413 100644
--- a/headroom/proxy/handlers/anthropic.py
+++ b/headroom/proxy/handlers/anthropic.py
@@ -1069,16 +1069,32 @@ class AnthropicHandlerMixin:
# session id (and its sticky CCR/memory tools, beta headers, and
# frozen-prefix state). This synthetic message only derives the id; it
# is never forwarded.
+ # Both the session id and the conversation lineage derive from the
+ # SAME original client bytes (pre security-scan/hook mutation,
+ # which may rewrite messages differently turn-to-turn — a
+ # turn-dependent rewrite of the leading system text would rotate
+ # the id mid-conversation, orphaning all session-sticky state).
+ # The synthetic system message folds the top-level system prompt
+ # into both derivations, so conversations sharing an explicit
+ # header id but differing in system prompt keep separate lineages.
system_prompt = body.get("system")
session_messages = (
- [{"role": "system", "content": system_prompt}, *messages]
+ [{"role": "system", "content": system_prompt}, *original_client_messages]
if system_prompt is not None
- else messages
+ else original_client_messages
)
session_id = self.session_tracker_store.compute_session_id(
request, model, session_messages
)
- prefix_tracker = self.session_tracker_store.get_or_create(session_id, "anthropic")
+ # Resolve the tracker by conversation lineage within the session id
+ # (#2085): one model + system prompt spans a Claude Code session and
+ # all its parallel subagents, so concurrent conversations share this
+ # fallback id — on one shared tracker their interleaved histories
+ # thrash the frozen-prefix state and the provider prompt cache is
+ # re-written on nearly every call.
+ prefix_tracker = self.session_tracker_store.resolve_tracker(
+ session_id, "anthropic", messages=session_messages
+ )
frozen_message_count = prefix_tracker.get_frozen_message_count()
# Idle gap since the previous turn's response, snapshotted at fetch
# (before get_or_create bumped the access clock). Forwarded to the
diff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.py
index 69ee8c369..86e25fa1e 100644
--- a/headroom/proxy/handlers/openai.py
+++ b/headroom/proxy/handlers/openai.py
@@ -2683,10 +2683,19 @@ class OpenAIHandlerMixin:
optimized_messages = messages
optimized_tokens = original_tokens
- # Get prefix cache tracker for this session
- openai_session_id = self.session_tracker_store.compute_session_id(request, model, messages)
- openai_prefix_tracker = self.session_tracker_store.get_or_create(
- openai_session_id, "openai"
+ # Get prefix cache tracker for this session, resolved by conversation
+ # lineage within the session id (#2085) so concurrent conversations
+ # sharing a fallback id (same model + system prompt) do not thrash one
+ # tracker's frozen-prefix state. Both the id and the lineage derive
+ # from the SAME original client bytes — pre image compression and
+ # pre_compress hook, both of which may rewrite history bytes
+ # differently turn-to-turn (a turn-dependent rewrite of the leading
+ # system text would rotate the id mid-conversation).
+ openai_session_id = self.session_tracker_store.compute_session_id(
+ request, model, original_client_messages
+ )
+ openai_prefix_tracker = self.session_tracker_store.resolve_tracker(
+ openai_session_id, "openai", messages=original_client_messages
)
# PR-A6 (P5-50, preps P0-6): session-sticky `OpenAI-Beta` merge.
@@ -5292,7 +5301,9 @@ class OpenAIHandlerMixin:
upstream: Any = None
from headroom.proxy.helpers import merge_extra_headers
- upstream_headers = merge_extra_headers(upstream_headers, self.config.openai_extra_headers)
+ upstream_headers = merge_extra_headers(
+ upstream_headers, self.config.openai_extra_headers
+ )
for ws_attempt in range(ws_connect_attempts):
try:
diff --git a/headroom/proxy/handlers/streaming.py b/headroom/proxy/handlers/streaming.py
index d4007e141..53a241e93 100644
--- a/headroom/proxy/handlers/streaming.py
+++ b/headroom/proxy/handlers/streaming.py
@@ -70,7 +70,12 @@ class StreamingMixin:
def _get_session_key(body: dict, session_header: str | None = None) -> str:
"""Return session identity from an explicit header or a body-derived hash.
- Fallback mirrors prefix_tracker.compute_session_id: md5(model:system[:500]).
+ The fallback is a coarse, self-contained hash (md5 of
+ ``model:system[:500]``) used ONLY to key the mid-turn steering state
+ below. It is NOT the same derivation as
+ ``prefix_tracker.compute_session_id`` (which json-encodes the full
+ leading system-text run) — do not key cross-subsystem state on the
+ assumption that the two fallbacks agree.
"""
if session_header:
return session_header
diff --git a/tests/test_anthropic_pre_upstream_backpressure.py b/tests/test_anthropic_pre_upstream_backpressure.py
index 805bb9ec0..3595fc429 100644
--- a/tests/test_anthropic_pre_upstream_backpressure.py
+++ b/tests/test_anthropic_pre_upstream_backpressure.py
@@ -172,6 +172,14 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):
update_from_response=lambda *a, **k: None,
record_request=lambda *a, **k: None,
),
+ resolve_tracker=lambda *a, **k: SimpleNamespace(
+ _cached_token_count=0,
+ get_frozen_message_count=lambda: 0,
+ get_last_original_messages=lambda: [],
+ get_last_forwarded_messages=lambda: [],
+ update_from_response=lambda *a, **k: None,
+ record_request=lambda *a, **k: None,
+ ),
)
# Unit 4: the only field this test cares about.
self.anthropic_pre_upstream_sem = anthropic_pre_upstream_sem
diff --git a/tests/test_anthropic_stage_timings.py b/tests/test_anthropic_stage_timings.py
index cd4177310..93b3e1944 100644
--- a/tests/test_anthropic_stage_timings.py
+++ b/tests/test_anthropic_stage_timings.py
@@ -102,6 +102,12 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):
get_last_forwarded_messages=lambda: [],
record_request=lambda *a, **k: None,
),
+ resolve_tracker=lambda *a, **k: SimpleNamespace(
+ get_frozen_message_count=lambda: 0,
+ get_last_original_messages=lambda: [],
+ get_last_forwarded_messages=lambda: [],
+ record_request=lambda *a, **k: None,
+ ),
)
async def _next_request_id(self) -> str:
diff --git a/tests/test_cache/test_prefix_tracker.py b/tests/test_cache/test_prefix_tracker.py
index 76af68bdd..69f39e59e 100644
--- a/tests/test_cache/test_prefix_tracker.py
+++ b/tests/test_cache/test_prefix_tracker.py
@@ -370,8 +370,8 @@ class TestSessionTrackerStore:
id3 = store.compute_session_id(MockRequest(), "gpt-4", messages)
assert id3 != id1
- def test_compute_session_id_uses_all_system_messages(self, store):
- """Different dynamic system messages should not collide."""
+ def test_compute_session_id_distinguishes_leading_system_run(self, store):
+ """Different dynamic LEADING system messages should not collide."""
class MockRequest:
headers = {}
@@ -454,6 +454,340 @@ class TestSessionTrackerStore:
assert isinstance(session_id, str)
assert len(session_id) == 16
+ def test_mid_conversation_system_turns_do_not_rotate_session_id(self, store):
+ """Claude Code sends <system-reminder> turns as role:"system" MESSAGES
+ interleaved into the history (hook outputs, skills lists, truncation
+ notices). Hashing those into the fallback id rotates the session id
+ mid-conversation — orphaning the prefix tracker and every other
+ session-sticky subsystem (beta headers, CCR/memory registries, the
+ compression cache) each time a reminder lands. Only the LEADING run of
+ system messages is session identity; later system turns are content.
+ """
+
+ class MockRequest:
+ headers = {}
+
+ leading = {"role": "system", "content": "You are an agent. " * 40}
+ turn1 = [leading, {"role": "user", "content": "read file A"}]
+ turn2 = turn1 + [
+ {"role": "assistant", "content": "read it"},
+ {"role": "user", "content": "tool result ..."},
+ {
+ "role": "system",
+ "content": "<system-reminder>Truncated: PARTIAL view</system-reminder>",
+ },
+ {"role": "user", "content": "continue"},
+ ]
+
+ id1 = store.compute_session_id(MockRequest(), "claude-sonnet-5", turn1)
+ id2 = store.compute_session_id(MockRequest(), "claude-sonnet-5", turn2)
+ assert id1 == id2
+
+
+class TestConversationLineageResolution:
+ """resolve_tracker: one PrefixCacheTracker per conversation lineage (#2085).
+
+ Concurrent conversations that share a fallback session id (same model +
+ same system prompt — e.g. a Claude Code session and its parallel subagents)
+ must not thrash one tracker's frozen-prefix state: interleaved turns would
+ each see the *other* conversation's prefix, freeze never stabilizes, and
+ the provider prompt cache is re-written on every call.
+
+ resolve_tracker keys trackers by message lineage instead: an incoming
+ history that extends a known lineage reuses its tracker; a diverging or
+ rewritten history gets a fresh one. The session id itself is never changed,
+ so session-sticky state keyed on it elsewhere (beta headers, CCR/memory
+ registries, the compression cache) is unaffected.
+ """
+
+ @pytest.fixture
+ def store(self):
+ return SessionTrackerStore()
+
+ @staticmethod
+ def _history(name: str, turn: int) -> list[dict]:
+ """Client-shaped request messages for `turn` (1-based): u0,a0,...,u_{turn-1}."""
+ messages: list[dict] = []
+ for t in range(turn):
+ messages.append({"role": "user", "content": f"[{name}] user {t} " + "x" * 200})
+ if t < turn - 1:
+ messages.append(
+ {"role": "assistant", "content": f"[{name}] assistant {t} " + "y" * 200}
+ )
+ return messages
+
+ @staticmethod
+ def _block_history(turn: int, cc_on: int | None) -> list[dict]:
+ """Block-content history; cache_control breakpoint on message `cc_on` (or none)."""
+ messages: list[dict] = []
+ for t in range(turn):
+ messages.append(
+ {"role": "user", "content": [{"type": "text", "text": f"user {t} " + "x" * 200}]}
+ )
+ if t < turn - 1:
+ messages.append(
+ {
+ "role": "assistant",
+ "content": [{"type": "text", "text": f"assistant {t} " + "y" * 200}],
+ }
+ )
+ if cc_on is not None:
+ msg = messages[cc_on]
+ blocks = [dict(b) for b in msg["content"]]
+ blocks[-1] = {**blocks[-1], "cache_control": {"type": "ephemeral"}}
+ messages[cc_on] = {**msg, "content": blocks}
+ return messages
+
+ def test_interleaved_conversations_resolve_to_independent_trackers(self, store):
+ """The #2085 production shape: two conversations, one session id,
+ alternating requests. Each must keep its own tracker and per-turn state
+ (on a shared tracker, _turn_number would count both conversations)."""
+ sid = "shared-fallback-id"
+ trackers: dict[str, PrefixCacheTracker] = {}
+ for turn in range(1, 5):
+ for name in ("A", "B"):
+ history = self._history(name, turn)
+ tracker = store.resolve_tracker(sid, "anthropic", messages=history)
+ trackers.setdefault(name, tracker)
+ assert tracker is trackers[name], f"[{name}] turn {turn} switched trackers"
+ tracker.update_from_response(
+ cache_read_tokens=1000 * turn,
+ cache_write_tokens=500,
+ messages=history,
+ )
+ assert trackers["A"] is not trackers["B"]
+ assert trackers["A"]._turn_number == 4
+ assert trackers["B"]._turn_number == 4
+
+ def test_identical_first_turns_share_until_divergence_then_split(self, store):
+ """Templated fan-outs send byte-identical first turns. While histories
+ are identical, sharing a tracker is harmless (the provider cache line
+ is identical too); they must split as soon as the histories diverge."""
+ sid = "shared"
+ first = [{"role": "user", "content": "verify the fix " + "p" * 300}]
+ t_a1 = store.resolve_tracker(sid, "anthropic", messages=first)
+ t_b1 = store.resolve_tracker(sid, "anthropic", messages=first)
+ assert t_b1 is t_a1
+
+ a2 = first + [
+ {"role": "assistant", "content": "answer A"},
+ {"role": "user", "content": "next A"},
+ ]
+ b2 = first + [
+ {"role": "assistant", "content": "answer B"},
+ {"role": "user", "content": "next B"},
+ ]
+ t_a2 = store.resolve_tracker(sid, "anthropic", messages=a2)
+ t_b2 = store.resolve_tracker(sid, "anthropic", messages=b2)
+ assert t_a2 is not t_b2
+
+ a3 = a2 + [
+ {"role": "assistant", "content": "answer A2"},
+ {"role": "user", "content": "next A2"},
+ ]
+ assert store.resolve_tracker(sid, "anthropic", messages=a3) is t_a2
+
+ @pytest.mark.parametrize(
+ "cc_turn2",
+ [0, -1, None],
+ ids=["breakpoint-stays", "breakpoint-moved-to-last", "breakpoint-removed"],
+ )
+ def test_cache_control_movement_does_not_split_lineage(self, store, cc_turn2):
+ """Clients move the cache_control breakpoint every turn; that must not
+ read as a rewritten history."""
+ sid = "shared"
+ t1 = store.resolve_tracker(sid, "anthropic", messages=self._block_history(1, cc_on=0))
+ t2 = store.resolve_tracker(
+ sid, "anthropic", messages=self._block_history(2, cc_on=cc_turn2)
+ )
+ assert t2 is t1
+
+ @pytest.mark.parametrize(
+ "requote",
+ [
+ lambda m: {**m, "content": [{"type": "text", "text": m["content"]}]},
+ lambda m: {
+ **m,
+ "content": [{"type": "text", "text": m["content"], "index": 0}],
+ },
+ lambda m: {
+ **m,
+ "content": [
+ {"type": "text", "text": m["content"]},
+ {"cachePoint": {"type": "default"}},
+ ],
+ },
+ ],
+ ids=["string-to-block-sugar", "streaming-index-annotation", "bedrock-cachepoint-block"],
+ )
+ def test_representation_churn_does_not_split_lineage(self, store, requote):
+ """Clients re-encode history turn-to-turn without changing content
+ (litellm flips string<->block sugar, streaming assembly adds `index`,
+ Bedrock moves its cachePoint block). Lineage matching must use the
+ same canonical equivalence as the cache-stable delta path."""
+ sid = "shared"
+ first = {"role": "user", "content": "hello " + "x" * 200}
+ t1 = store.resolve_tracker(sid, "anthropic", messages=[first])
+ grown = [
+ requote(first),
+ {"role": "assistant", "content": "hi"},
+ {"role": "user", "content": "next"},
+ ]
+ assert store.resolve_tracker(sid, "anthropic", messages=grown) is t1
+
+ @pytest.mark.parametrize(
+ "rewrite",
+ [
+ lambda h: [{"role": "user", "content": "[summary of the conversation so far]"}],
+ lambda h: [h[0], {"role": "assistant", "content": "EDITED"}, *h[2:]],
+ lambda h: h[:-2],
+ ],
+ ids=["compacted", "middle-edited", "truncated"],
+ )
+ def test_rewritten_history_gets_fresh_tracker(self, store, rewrite):
+ """A rewritten history (client-side /compact, edits, truncation) means
+ the provider cache line is gone anyway: start a fresh lineage, keep the
+ old tracker until TTL, and never touch the session id."""
+ sid = "shared"
+ history = self._history("A", 3)
+ tracker = store.resolve_tracker(sid, "anthropic", messages=history)
+ fresh = store.resolve_tracker(sid, "anthropic", messages=rewrite(history))
+ assert fresh is not tracker
+ assert store.active_sessions == 2
+
+ @pytest.mark.parametrize("messages", [None, []], ids=["none", "empty"])
+ def test_resolve_without_messages_matches_legacy_get_or_create(self, store, messages):
+ tracker = store.resolve_tracker("sid", "anthropic", messages=messages)
+ assert tracker is store.get_or_create("sid", "anthropic")
+
+ def test_resolve_with_freeze_disabled_matches_legacy_get_or_create(self):
+ """With prefix freeze off there is no frozen state to protect — skip
+ lineage bookkeeping entirely."""
+ store = SessionTrackerStore(PrefixFreezeConfig(enabled=False))
+ t_a = store.resolve_tracker("sid", "anthropic", messages=self._history("A", 1))
+ t_b = store.resolve_tracker("sid", "anthropic", messages=self._history("B", 1))
+ assert t_a is t_b
+ assert t_a is store.get_or_create("sid", "anthropic")
+
+ def test_over_cap_conversations_share_one_overflow_tracker(self):
+ """A fan-out storm on one session id must not grow trackers unbounded:
+ past the cap, new conversations share one overflow tracker instead of
+ evicting an established lineage."""
+ store = SessionTrackerStore(PrefixFreezeConfig(max_lineages_per_session=4))
+ sid = "storm"
+ overflow = set()
+ for i in range(9):
+ tracker = store.resolve_tracker(
+ sid, "anthropic", messages=[{"role": "user", "content": f"task {i} " + "z" * 100}]
+ )
+ if i >= 4:
+ overflow.add(id(tracker))
+ assert store.active_sessions == 5 # 4 lineages + 1 shared overflow
+ assert len(overflow) == 1
+
+ def test_established_lineages_survive_cap_overflow(self):
+ """Filling the family must never evict an established conversation —
+ under round-robin any eviction victim is the next requester, which
+ would degrade EVERY conversation to a cold tracker per turn."""
+ store = SessionTrackerStore(PrefixFreezeConfig(max_lineages_per_session=2))
+ sid = "shared"
+ parent_history = self._history("parent", 4)
+ parent = store.resolve_tracker(sid, "anthropic", messages=parent_history)
+ shorty = store.resolve_tracker(sid, "anthropic", messages=self._history("shorty", 1))
+ # Third divergent conversation lands on the overflow tracker; both
+ # established lineages keep their trackers.
+ newcomer = store.resolve_tracker(sid, "anthropic", messages=self._history("new", 1))
+ assert newcomer is not parent
+ assert newcomer is not shorty
+ grown = parent_history + [
+ {"role": "assistant", "content": "[parent] assistant 3 " + "y" * 200},
+ {"role": "user", "content": "[parent] user 4 " + "x" * 200},
+ ]
+ assert store.resolve_tracker(sid, "anthropic", messages=grown) is parent
+ assert (
+ store.resolve_tracker(sid, "anthropic", messages=self._history("shorty", 2)) is shorty
+ )
+
+ def test_no_cliff_at_cap_plus_one_round_robin(self):
+ """cap+1 conversations round-robining turns: the in-cap conversations
+ keep their trackers on every round (no eviction churn); only the
+ over-cap tail shares the overflow tracker."""
+ store = SessionTrackerStore(PrefixFreezeConfig(max_lineages_per_session=3))
+ sid = "shared"
+ trackers: dict[str, PrefixCacheTracker] = {}
+ for turn in range(1, 4):
+ for name in ("A", "B", "C", "D"):
+ tracker = store.resolve_tracker(
+ sid, "anthropic", messages=self._history(name, turn)
+ )
+ if turn == 1:
+ trackers[name] = tracker
+ elif name != "D":
+ assert tracker is trackers[name], f"[{name}] turn {turn} lost its tracker"
+ else:
+ assert tracker is trackers["D"] # stable overflow tracker
+
+ def test_empty_canonical_history_falls_back_to_legacy(self):
+ """A history whose every message projects away (pure directive
+ content) carries no lineage signal — behave like get_or_create."""
+ store = SessionTrackerStore()
+ tracker = store.resolve_tracker("sid", "anthropic", messages=[{}])
+ assert tracker is store.get_or_create("sid", "anthropic")
+
+ def test_nan_in_tool_payload_does_not_split_lineage(self):
+ """json.loads accepts bare NaN, and NaN != NaN — a resent history
+ containing one must still read as the same conversation."""
+ store = SessionTrackerStore()
+ sid = "shared"
+ turn1 = [
+ {"role": "user", "content": "run the tool " + "x" * 200},
+ {
+ "role": "assistant",
+ "content": [
+ {"type": "tool_use", "id": "t1", "name": "score", "input": {"v": float("nan")}}
+ ],
+ },
+ {
+ "role": "user",
+ "content": [{"type": "tool_result", "tool_use_id": "t1", "content": "ok"}],
+ },
+ ]
+ t1 = store.resolve_tracker(sid, "anthropic", messages=turn1)
+ turn2 = turn1 + [
+ {"role": "assistant", "content": "done"},
+ {"role": "user", "content": "next"},
+ ]
+ assert store.resolve_tracker(sid, "anthropic", messages=turn2) is t1
+
+ def test_expired_lineages_are_cleaned_up(self):
+ config = PrefixFreezeConfig(session_ttl_seconds=1)
+ store = SessionTrackerStore(default_config=config)
+ for name in ("A", "B"):
+ tracker = store.resolve_tracker("sid", "anthropic", messages=self._history(name, 1))
+ tracker._last_activity = time.time() - 2
+
+ store._last_cleanup = 0
+ store._maybe_cleanup()
+ assert store.active_sessions == 0
+
+ # The lineage index must not resurrect evicted trackers: extending an
+ # evicted conversation starts cold.
+ fresh = store.resolve_tracker("sid", "anthropic", messages=self._history("A", 2))
+ assert fresh._turn_number == 0
+
+ def test_shared_session_id_is_not_rotated(self, store):
+ """Composition guard: lineage resolution must not leak into session-id
+ derivation — beta stickiness and the compression cache key on it."""
+
+ class MockRequest:
+ headers = {}
+
+ msgs_a = [{"role": "system", "content": "S"}, *self._history("A", 1)]
+ msgs_b = [{"role": "system", "content": "S"}, *self._history("B", 1)]
+ id_a = store.compute_session_id(MockRequest(), "claude-3", msgs_a)
+ id_b = store.compute_session_id(MockRequest(), "claude-3", msgs_b)
+ assert id_a == id_b
+
class TestMultiTurnScenario:
"""Integration-style tests simulating multi-turn conversations."""
diff --git a/tests/test_cold_start_fast_pass.py b/tests/test_cold_start_fast_pass.py
index 6c6baa4c6..a230facb4 100644
--- a/tests/test_cold_start_fast_pass.py
+++ b/tests/test_cold_start_fast_pass.py
@@ -139,6 +139,12 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):
get_last_forwarded_messages=lambda: [],
record_request=lambda *a, **k: None,
),
+ resolve_tracker=lambda *a, **k: SimpleNamespace(
+ get_frozen_message_count=lambda: 0,
+ get_last_original_messages=lambda: [],
+ get_last_forwarded_messages=lambda: [],
+ record_request=lambda *a, **k: None,
+ ),
)
# Cold-start deferral wiring under test.
self._background_compression_enabled = True