Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(proxy): gateway turn fixes found by benchmarking behind a gateway (#3619)
Tejas Chopraaceff2ea9dbd
diff --git a/docs/content/docs/proxy.mdx b/docs/content/docs/proxy.mdxindex d077a6ce8..8e2a0ee27 100644--- a/docs/content/docs/proxy.mdx+++ b/docs/content/docs/proxy.mdx@@ -685,7 +685,7 @@ The response half. Same exposure rules as `/v1/compress`: loopback-only unless `| `latency_ms` | number | no | Gateway-measured provider latency, added to the turn's recorded latency. Must be non-negative. || `round` | integer | no | Which provider call this response answers: `0` for the original forward, `n` for the `n`-th re-drive (the `round` this endpoint returned). Makes the call idempotent: a retried relay for a round already answered gets the same `redrive` answer again instead of being fed to the hook, and its usage is not billed twice. A round ahead of the turn is a `400`. || `usage` | object | no | The provider's usage block, in any accepted shape (below). |-| `response` | object | no | The full provider response JSON. **Required** when the turn carries `redrive`. |+| `response` | object | no | The full provider response JSON. **Required** when the turn carries `redrive` and the provider call succeeded (`2xx`). A failed call closes the turn with its `status` alone. || `model` | string | no | Provider-reported model. Accepted, not read today. |Accepted `usage` shapes — the whole provider response body works too, since a nested `usage` key is descended into once:@@ -706,11 +706,12 @@ There are two answers. `done` ends the turn:"response": null,"frozen_message_count": 12,"usage_applied": true,- "rounds": 0+ "rounds": 0,+ "billed_usage": { "input_tokens": 3120, "output_tokens": 212, "cache_read_input_tokens": 20414 }}```-`response` is a replacement to send to the client when a hook produced one; otherwise it is `null` and you forward the provider response you already hold. `frozen_message_count` is the session's count after the usage was applied (`null` for a stateless turn). `usage_applied` is `false` when the usage carried no cache signal, the session had expired, or its lock was busy — never an error, since the turn is finishing and there is nothing to retry with. After `done` the turn is gone.+`response` is what to send to the client. On a turn with no re-drive it is the hook's replacement when there is one, otherwise `null` and you forward the provider response you already hold. After one or more re-drives it is never `null`: it is the hook's replacement or the latest provider response, with `usage` set to the whole turn's bill. `frozen_message_count` is the session's count after the usage was applied (`null` for a stateless turn). `usage_applied` is `false` when the usage carried no cache signal, the session had expired, or its lock was busy — never an error, since the turn is finishing and there is nothing to retry with. `billed_usage` is what the provider billed for the whole turn: every counter, cache reads and writes included, summed across all rounds, in Anthropic keys (`null` when no usage was relayed). That total is the re-driven `response`'s `usage`, since the client made one call. After `done` the turn is gone.`redrive` asks for another provider call:@@ -723,7 +724,7 @@ There are two answers. `done` ends the turn:}```-Send `request` to the provider exactly as given — it is the provider body with the hook's messages and possibly reloaded tools — and post the provider's JSON back under the same `turn_id`. Past `HEADROOM_GATEWAY_MAX_REDRIVES` the turn ends with `done` and `response: null`. Hook failures never fail the turn either: the answer is `done` with `response: null` and the gateway forwards what it holds.+Send `request` to the provider exactly as given — it is the provider body with the hook's messages and possibly reloaded tools — and post the provider's JSON back under the same `turn_id`. Past `HEADROOM_GATEWAY_MAX_REDRIVES` the turn ends with `done` and the latest provider response. Hook failures never fail the turn either: the answer is `done`, with the latest provider response after a re-drive, or `response: null` before one, and the gateway forwards what it gets.| Status | Body | When || --- | --- | --- |diff --git a/headroom/proxy/gateway_extension.py b/headroom/proxy/gateway_extension.pyindex 6eef84ca8..1bbac9cd9 100644--- a/headroom/proxy/gateway_extension.py+++ b/headroom/proxy/gateway_extension.py@@ -187,6 +187,27 @@ class GatewayTurn:input_tokens=tokens_before,transforms=transforms,)+ if self.provider == "anthropic":+ # Last stop before the body leaves: a request over Anthropic's+ # cache_control budget is a guaranteed 400 on the provider.+ from headroom.proxy.helpers import enforce_cache_breakpoint_budget++ provider_body = self._fields["body"]+ client_messages = self.body.get("messages") if isinstance(self.body, dict) else None+ system, guarded, tools, budget = enforce_cache_breakpoint_budget(+ provider_body.get("system"),+ provider_body.get("messages"),+ provider_body.get("tools"),+ client_messages=client_messages,+ request_id=self._turn_id or "",+ )+ if budget["repaired"]:+ if provider_body.get("system") is not None:+ provider_body["system"] = system+ provider_body["messages"] = guarded+ if provider_body.get("tools") is not None:+ provider_body["tools"] = tools+ transforms.append("cache_breakpoint_budget")# I-BODY: top-level `messages` and `body.messages` are one list.return FinishedTurn(fields=self._fields,diff --git a/headroom/proxy/gateway_turn.py b/headroom/proxy/gateway_turn.pyindex b1899f5c4..851a03cae 100644--- a/headroom/proxy/gateway_turn.py+++ b/headroom/proxy/gateway_turn.py@@ -263,6 +263,37 @@ class NormalizedUsage:cache_write=other.cache_write if other.has_cache_signal else self.cache_write,)+ def summed_with(self, other: NormalizedUsage) -> NormalizedUsage:+ """Sum every counter across re-drive rounds: what the provider billed+ for the whole turn, cache reads and writes of every round included.+ ``merged_with`` keeps the latest cache state for freeze decisions;+ this is the bill."""++ def _sum(a: int | None, b: int | None) -> int | None:+ if a is None and b is None:+ return None+ return (a or 0) + (b or 0)++ return NormalizedUsage(+ input_tokens=_sum(self.input_tokens, other.input_tokens),+ output_tokens=_sum(self.output_tokens, other.output_tokens),+ cache_read=_sum(self.cache_read, other.cache_read),+ cache_write=_sum(self.cache_write, other.cache_write),+ )++ def as_anthropic(self) -> dict[str, int]:+ """Anthropic ``usage`` keys, only for the counters that were reported."""+ out: dict[str, int] = {}+ if self.input_tokens is not None:+ out["input_tokens"] = self.input_tokens+ if self.output_tokens is not None:+ out["output_tokens"] = self.output_tokens+ if self.cache_read is not None:+ out["cache_read_input_tokens"] = self.cache_read+ if self.cache_write is not None:+ out["cache_creation_input_tokens"] = self.cache_write+ return out+def _usage_int(usage: dict[str, Any], name: str) -> int | None:if name not in usage:@@ -475,8 +506,11 @@ class PendingTurn:runner: SuspendedHookRunner | None = Nonetags: dict[str, Any] = field(default_factory=dict)client: str | None = None- # Billed usage accumulated across every response-half call for the turn.+ # Usage accumulated across every response-half call for the turn: input+ # and output summed, cache fields from the latest round (freeze state).usage: NormalizedUsage | None = None+ # Every counter summed across rounds: the turn's bill.+ billed: NormalizedUsage | None = None# A second response-half call while one is being driven would race the# parked coroutine; the handler answers 409 instead.in_flight: bool = False@@ -1325,6 +1359,31 @@ def _redrive_payload(turn: PendingTurn, step: Step) -> dict[str, Any]:}+def _usage_in_shape(existing: dict[str, Any], billed: NormalizedUsage) -> dict[str, Any]:+ """Overwrite ``existing`` usage counters with ``billed``, keeping its shape.++ An OpenAI chat usage block (``prompt_tokens``) gets chat keys; anything+ else gets Anthropic keys. Counters the provider never reported stay out.+ """+ out = dict(existing)+ if "prompt_tokens" in existing or "completion_tokens" in existing:+ if billed.input_tokens is not None:+ out["prompt_tokens"] = billed.input_tokens+ if billed.output_tokens is not None:+ out["completion_tokens"] = billed.output_tokens+ if billed.input_tokens is not None or billed.output_tokens is not None:+ out["total_tokens"] = (billed.input_tokens or 0) + (billed.output_tokens or 0)+ if billed.cache_read is not None:+ details = existing.get("prompt_tokens_details")+ out["prompt_tokens_details"] = {+ **(details if isinstance(details, dict) else {}),+ "cached_tokens": billed.cache_read,+ }+ return out+ out.update(billed.as_anthropic())+ return out++async def handle_compress_response(proxy: Any, request: Any) -> Any:"""``POST /v1/compress/response`` — finish or re-drive a pending turn."""from fastapi.responses import JSONResponse@@ -1375,7 +1434,9 @@ async def handle_compress_response(proxy: Any, request: Any) -> Any:"unknown_turn",f"No pending turn {turn_id!r} (never registered, already finished, or expired).",)- if OBLIGATION_REDRIVE in turn.obligations and response is None:+ # A failed provider call has nothing to re-drive: its status (and usage,+ # if any) closes the turn. Only a successful response must carry the body.+ if OBLIGATION_REDRIVE in turn.obligations and response is None and 200 <= status < 300:return _error(400,"missing_response",@@ -1416,6 +1477,7 @@ async def handle_compress_response(proxy: Any, request: Any) -> Any:try:if usage is not None:turn.usage = usage if turn.usage is None else turn.usage.merged_with(usage)+ turn.billed = usage if turn.billed is None else turn.billed.summed_with(usage)step: Step | None = Noneif turn.hooks_armed and turn.ctx is not None:@@ -1461,7 +1523,10 @@ async def handle_compress_response(proxy: Any, request: Any) -> Any:frozen, applied = await apply_session_usage(proxy, turn, turn.usage)if turn.outcome_draft is not None:outcome = complete_outcome(- turn.outcome_draft, turn.usage, status=status, latency_ms=latency_ms+ turn.outcome_draft,+ turn.billed or turn.usage,+ status=status,+ latency_ms=latency_ms,)turn.outcome_draft = Noneawait proxy._record_request_outcome(outcome)@@ -1470,6 +1535,25 @@ async def handle_compress_response(proxy: Any, request: Any) -> Any:)if registry is not None:registry.pop(turn_id)+ billed = turn.billed.as_anthropic() if turn.billed is not None else None+ # After a re-drive the answer is either the hook's replacement or, when+ # the hook handed back the latest provider response (``response: null``),+ # the response the gateway posted on this call. Either way the client made+ # one call, so its response reports what the provider billed for all of+ # it, the way a server-side tool loop does, in the response's own usage+ # shape. A turn with no re-drive keeps ``response: null``: the gateway's+ # held response already carries the whole bill.+ answer = final_response if final_response is not None else response+ if (+ turn.rounds > 0+ and billed+ and isinstance(answer, dict)+ and isinstance(answer.get("usage"), dict)+ ):+ final_response = {+ **answer,+ "usage": _usage_in_shape(answer["usage"], turn.billed),+ }return JSONResponse({"action": "done",@@ -1478,6 +1562,8 @@ async def handle_compress_response(proxy: Any, request: Any) -> Any:"frozen_message_count": frozen,"usage_applied": applied,"rounds": turn.rounds,+ # Every counter summed across rounds, Anthropic keys.+ "billed_usage": billed,})finally:diff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.pyindex b6e8cf36e..cb7078e54 100644--- a/headroom/proxy/handlers/openai.py+++ b/headroom/proxy/handlers/openai.py@@ -10113,7 +10113,17 @@ class OpenAIHandlerMixin:# the pipeline introduced — byte-identical is the contract# the caller forwards on.turn = finalize_turn(result.messages, messages, prev_original, prev_returned)- final = turn.messages+ # A replay can carry an earlier turn's cache_control marker+ # after the caller moved its breakpoint forward, and+ # Anthropic rejects more than four. Keep message markers at+ # the caller's current positions, as the /v1/messages+ # handler does after its own replay. The gateway's finish+ # step still enforces the hard budget on the whole body.+ from headroom.cache.prefix_tracker import normalize_message_cache_control++ final = normalize_message_cache_control(+ turn.messages, prev_returned, client_messages=messages+ )if _turn is not None:# AFTER the replay overlay, BEFORE update_from_result /# record_returned: the turn's output is what the callerdiff --git a/headroom/proxy/helpers.py b/headroom/proxy/helpers.pyindex 2d4f1d700..8a33186fe 100644--- a/headroom/proxy/helpers.py+++ b/headroom/proxy/helpers.py@@ -740,6 +740,129 @@ def enforce_cache_control_ttl_order(return system, messages, tools, stats+#: Anthropic rejects a request with more than this many cache_control blocks+#: across tools, system and messages ("A maximum of 4 blocks with+#: cache_control may be provided").+ANTHROPIC_MAX_CACHE_BREAKPOINTS = 4+++def enforce_cache_breakpoint_budget(+ system: Any,+ messages: Any,+ tools: Any,+ *,+ client_messages: Any = None,+ limit: int = ANTHROPIC_MAX_CACHE_BREAKPOINTS,+ request_id: str = "",+) -> tuple[Any, Any, Any, dict[str, Any]]:+ """Last-stop guard: never forward more ``cache_control`` blocks than allowed.++ A request over the limit is a guaranteed 400, and the client cannot have+ caused it with its own markers, so whatever pushed it over was added on+ the way through. Repairs in the order that costs the least caching:++ 1. Re-place message markers at the client's own positions+ (:func:`~headroom.cache.prefix_tracker.normalize_message_cache_control`),+ which drops markers a replay or transform carried in.+ 2. If still over, drop the OLDEST message markers first: the newest one is+ the client's write anchor for the growing tail, and system/tools+ markers cover the hottest, longest-lived prefix.+ 3. If still over, drop system markers, then tools markers, earliest first.++ Returns ``(system, messages, tools, stats)``; the inputs come back by+ identity when nothing needed repairing. Logs one WARNING per repair with+ the per-section counts so the source of the extra marker is traceable.+ """+ before = count_cache_breakpoints(system, messages, tools)+ stats: dict[str, Any] = {"repaired": False, "before": before, "after": before}+ if before["total"] <= limit:+ return system, messages, tools, stats++ if isinstance(messages, list) and client_messages is not None:+ from headroom.cache.prefix_tracker import normalize_message_cache_control++ messages = normalize_message_cache_control(messages, None, client_messages=client_messages)++ for section in ("messages", "system", "tools"):+ excess = count_cache_breakpoints(system, messages, tools)["total"] - limit+ if excess <= 0:+ break+ budget = {"left": excess}+ # Removal, oldest first in Anthropic's evaluation order. Copy-on-write:+ # the forwarded body shares structure with the session snapshot.+ if section == "messages" and isinstance(messages, list):+ rebuilt: list[Any] = []+ for msg in messages:+ if budget["left"] <= 0 or not isinstance(msg, dict):+ rebuilt.append(msg)+ continue+ content = msg.get("content")+ if isinstance(content, list):+ new_blocks = []+ for block in content:+ if budget["left"] > 0 and isinstance(block, dict):+ inner = block.get("content")+ if isinstance(inner, list):+ new_inner = []+ for sub in inner:+ if (+ budget["left"] > 0+ and isinstance(sub, dict)+ and "cache_control" in sub+ ):+ sub = {k: v for k, v in sub.items() if k != "cache_control"}+ budget["left"] -= 1+ new_inner.append(sub)+ block = {**block, "content": new_inner}+ if budget["left"] > 0 and "cache_control" in block:+ block = {k: v for k, v in block.items() if k != "cache_control"}+ budget["left"] -= 1+ new_blocks.append(block)+ msg = {**msg, "content": new_blocks}+ if budget["left"] > 0 and "cache_control" in msg:+ msg = {k: v for k, v in msg.items() if k != "cache_control"}+ budget["left"] -= 1+ rebuilt.append(msg)+ messages = rebuilt+ elif section in ("system", "tools"):+ holders = system if section == "system" else tools+ if isinstance(holders, list):+ rebuilt_h = []+ for holder in holders:+ if (+ budget["left"] > 0+ and isinstance(holder, dict)+ and "cache_control" in holder+ ):+ holder = {k: v for k, v in holder.items() if k != "cache_control"}+ budget["left"] -= 1+ rebuilt_h.append(holder)+ if section == "system":+ system = rebuilt_h+ else:+ tools = rebuilt_h++ after = count_cache_breakpoints(system, messages, tools)+ stats.update({"repaired": True, "after": after})+ logger.warning(+ "event=cache_breakpoint_budget request_id=%s limit=%d "+ "before_total=%d before_system=%d before_tools=%d before_messages=%d "+ "after_total=%d after_system=%d after_tools=%d after_messages=%d; "+ "the outbound request carried more cache_control blocks than the provider accepts",+ request_id,+ limit,+ before["total"],+ before["system"],+ before["tools"],+ before["messages"],+ after["total"],+ after["system"],+ after["tools"],+ after["messages"],+ )+ return system, messages, tools, stats++def log_cache_breakpoints(*,request_id: str | None,diff --git a/headroom/proxy/server.py b/headroom/proxy/server.pyindex 9cd7694d7..1e690a972 100644--- a/headroom/proxy/server.py+++ b/headroom/proxy/server.py@@ -1873,6 +1873,75 @@ class HeadroomProxy(return eager_status, transform_statuses+ def _start_kompress_background_warmup(self) -> bool:+ """Load the Kompress model on a daemon thread once startup has returned.++ Startup must not build the model (native init before the port binds+ segfaults on RHEL/CentOS 7-family hosts, #1908), so it used to load+ inside the first request that needed it: a benchmark turn through a+ gateway stalled 21 s while transformers, torch and the ONNX session+ came up. Loading on a background thread after a short delay moves that+ cost off the request path. Skipped on glibc older than 2.28 (the+ affected host family) and when ``HEADROOM_KOMPRESS_WARMUP`` is ``0``;+ ``1`` forces it. Returns ``True`` when a warm-up thread was started.+ """+ raw = os.environ.get("HEADROOM_KOMPRESS_WARMUP", "").strip().lower()+ if raw in ("0", "false", "no", "off"):+ return False+ if not raw and os.environ.get("PYTEST_CURRENT_TEST"):+ # Test apps boot by the hundred; none of them should load a model.+ return False+ if raw not in ("1", "true", "yes", "on"):+ import platform++ libc, version = platform.libc_ver()+ if libc == "glibc":+ try:+ major, minor = (int(part) for part in version.split(".")[:2])+ except ValueError:+ major, minor = 0, 0+ if (major, minor) < (2, 28):+ return False+ compressor = None+ for pipeline in (self.anthropic_pipeline, self.openai_pipeline):+ for transform in getattr(pipeline, "transforms", []):+ getter = getattr(transform, "_get_kompress", None)+ if getter is None:+ continue+ try:+ compressor = getter()+ except Exception:+ compressor = None+ if compressor is not None and hasattr(compressor, "preload"):+ break+ compressor = None+ if compressor is not None:+ break+ if compressor is None:+ return False++ try:+ delay = float(os.environ.get("HEADROOM_KOMPRESS_WARMUP_DELAY_SECONDS", "2") or 2)+ except ValueError:+ delay = 2.0++ def _warm() -> None:+ time.sleep(max(0.0, delay))+ started = time.monotonic()+ try:+ backend = compressor.preload(allow_download=True)+ except Exception as exc: # the lazy request path still loads on first use+ logger.warning("Kompress background warm-up failed: %s", exc)+ return+ logger.info(+ "Kompress: warmed in the background in %.0f ms (backend %s)",+ (time.monotonic() - started) * 1000,+ backend,+ )++ threading.Thread(target=_warm, name="kompress-warmup", daemon=True).start()+ return True+async def startup(self):"""Initialize async resources."""self._get_shutdown_event().clear()@@ -2013,7 +2082,10 @@ class HeadroomProxy(if self._kompress_status == "enabled":logger.info("Kompress: ENABLED (ModernBERT token compressor)")elif self._kompress_status == "deferred":- logger.info("Kompress: DEFERRED (model loads on first request)")+ if self._start_kompress_background_warmup():+ logger.info("Kompress: DEFERRED (warming in the background after startup)")+ else:+ logger.info("Kompress: DEFERRED (model loads on first request)")elif self.config.optimize:logger.info("Kompress: not installed (pip install headroom-ai[ml] for ML compression)")diff --git a/headroom/transforms/content_router.py b/headroom/transforms/content_router.pyindex 9734db464..207650681 100644--- a/headroom/transforms/content_router.py+++ b/headroom/transforms/content_router.py@@ -617,6 +617,23 @@ def _fenced_shell_command(content: Any) -> str:return m.group(1).strip() if m else ""+def _answers_fenced_command(messages: list[dict[str, Any]], index: int) -> bool:+ """True when the user message at ``index`` replies to a fenced shell command.++ Text-based harnesses (mini-swe-agent and similar) send a command in a fenced+ block in the assistant turn and return its output as the next plain user+ message. That message is a tool observation, not the caller's prompt. Walks+ back to the nearest assistant turn, stopping at an earlier user turn.+ """+ for j in range(index - 1, -1, -1):+ role = messages[j].get("role") if isinstance(messages[j], dict) else None+ if role == "assistant":+ return bool(_fenced_shell_command(messages[j].get("content")))+ if role == "user":+ return False+ return False++_READ_VERBS = ("cat", "head", "tail", "nl", "bat", "less", "more")# Machine-generated dependency lockfiles detect as PLAIN_TEXT (so the content-based@@ -4973,6 +4990,16 @@ class ContentRouter(Transform):# --- Adaptive parameters based on context pressure ---num_messages = len(messages)+ # The opening prompt: every user message before the first assistant+ # turn. See `protect_prompt_text` in _process_content_blocks.+ first_assistant_index = next(+ (+ idx+ for idx, msg in enumerate(messages)+ if isinstance(msg, dict) and msg.get("role") == "assistant"+ ),+ num_messages,+ )model_limit = kwargs.get("model_limit", 0)# Adaptive Read protection: protect a fraction of recent messages@@ -5177,6 +5204,23 @@ class ContentRouter(Transform):bias = 1.0 # Default bias, may be overridden for tool messagesmessages_from_end = num_messages - i+ # The caller's own words stay verbatim on a replaying path even+ # when user messages are compressible for their tool observations:+ # the opening prompt (a task statement with test ids and paths)+ # and the text of the newest user turn. Lossy text compression+ # there rewrites what the model is asked to do, and nothing but a+ # cache_control marker used to stop it -- a marker Claude Code sets+ # and plain agents do not. The one newest user turn that is not a+ # prompt is a text harness's tool observation: the reply to a+ # fenced shell command in the assistant turn before it.+ prompt_turn = (+ prefix_replay_guaranteed+ and role == "user"+ and (+ i < first_assistant_index+ or (messages_from_end == 1 and not _answers_fenced_command(messages, i))+ )+ )# Handle list content (Anthropic format with content blocks)if isinstance(content, list):@@ -5199,6 +5243,7 @@ class ContentRouter(Transform):skip_system=skip_system,compress_assistant_text_blocks=compress_assistant_text_blocks,prefix_replay_guaranteed=prefix_replay_guaranteed,+ protect_prompt_text=prompt_turn,)result_slots[i] = transformed_messageroute_counts["content_blocks"] += 1@@ -5315,8 +5360,9 @@ class ContentRouter(Transform):route_counts["read_protected"] += 1continue- # Protection 1: Never compress user messages (unless overridden)- if skip_user and role == "user":+ # Protection 1: Never compress user messages (unless overridden),+ # and never the caller's prompt on a replaying path.+ if role == "user" and (skip_user or prompt_turn):result_slots[i] = messagetransforms_applied.append("router:protected:user_message")route_counts["user_msg"] += 1@@ -6089,6 +6135,7 @@ class ContentRouter(Transform):skip_system: bool = True,compress_assistant_text_blocks: bool = False,prefix_replay_guaranteed: bool = False,+ protect_prompt_text: bool = False,) -> dict[str, Any]:"""Process content blocks (Anthropic format) for compression.@@ -6150,6 +6197,10 @@ class ContentRouter(Transform):skip_system: If True, never compress text blocks in system-role messages.compress_assistant_text_blocks: If True, allow compressing text blocks inassistant-role messages. Default False (cache-safe).+ protect_prompt_text: If True, text blocks in this user message are the+ caller's prompt (the opening task or the newest user turn) and stay+ verbatim even when ``skip_user`` is False; tool_result blocks in the+ same message are still compressible.Returns:Transformed message with compressed content blocks.@@ -6162,7 +6213,7 @@ class ContentRouter(Transform):# outputs and compress freely; assistant defaults to skip (cache# safety) with explicit opt-in; unknown roles default to skip.if role == "user":- protect_text_blocks = skip_user+ protect_text_blocks = skip_user or protect_prompt_textelif role in {"system", "developer"}:protect_text_blocks = skip_systemelif role == "assistant":diff --git a/tests/gateway/test_cache_breakpoint_budget.py b/tests/gateway/test_cache_breakpoint_budget.pynew file mode 100644index 000000000..0c1b43aeb--- /dev/null+++ b/tests/gateway/test_cache_breakpoint_budget.py@@ -0,0 +1,109 @@+"""The gateway body never carries more cache_control blocks than Anthropic accepts.++A LiteLLM benchmark run saw ``A maximum of 4 blocks with cache_control may be+provided. Found 5.`` on a turn whose only Headroom work was tool-result+compression and tool schema compaction. The client (Claude Code) never sends+more than four, so the fifth was added on the way through. Whatever the path,+the last stop before the body leaves must hold the budget.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import Any++from headroom.proxy.helpers import count_cache_breakpoints, enforce_cache_breakpoint_budget+from headroom.proxy.turn_hooks import register_turn_hook+from tests.gateway.conftest import compress++CC = {"type": "ephemeral"}+++def _system(markers: int) -> list[dict[str, Any]]:+ return [+ {+ "type": "text",+ "text": f"system part {i}",+ **({"cache_control": CC} if i < markers else {}),+ }+ for i in range(2)+ ]+++def _messages(marked: set[int], n: int = 6) -> list[dict[str, Any]]:+ out = []+ for i in range(n):+ block = {"type": "text", "text": f"message {i}"}+ if i in marked:+ block["cache_control"] = CC+ out.append({"role": "user" if i % 2 == 0 else "assistant", "content": [block]})+ return out+++def test_under_budget_is_returned_by_identity() -> None:+ system, messages = _system(2), _messages({4, 5})+ s, m, t, stats = enforce_cache_breakpoint_budget(system, messages, None)+ assert (s, m, t) == (system, messages, None)+ assert s is system and m is messages+ assert stats["repaired"] is False+++def test_extra_message_marker_goes_back_to_client_positions() -> None:+ client = _messages({4, 5})+ outbound = _messages({2, 4, 5}) # one marker a replay or transform carried in+ s, m, _, stats = enforce_cache_breakpoint_budget(+ _system(2), outbound, None, client_messages=client+ )+ assert stats["repaired"] is True+ assert count_cache_breakpoints(s, m, None)["total"] == 4+ marked = [i for i, msg in enumerate(m) if "cache_control" in msg["content"][0]]+ assert marked == [4, 5]+++def test_without_client_positions_the_oldest_message_markers_go_first() -> None:+ outbound = _messages({1, 3, 5})+ s, m, _, stats = enforce_cache_breakpoint_budget(_system(2), outbound, None)+ assert count_cache_breakpoints(s, m, None)["total"] == 4+ marked = [i for i, msg in enumerate(m) if "cache_control" in msg["content"][0]]+ # The newest marker is the write anchor for the growing tail: it stays.+ assert marked == [3, 5]+ assert all("cache_control" in b for b in s)+ # Copy-on-write: the caller's objects are untouched.+ assert "cache_control" in outbound[1]["content"][0]+++@dataclass+class _MarkerLeakHook:+ """Stands in for any stage that leaves an extra marker on an old message."""++ name: str = "test_marker_leak"+ savings_source: str = "test_marker_leak"+ stream_safe: bool = True++ def on_request(self, ctx: Any) -> None:+ messages = [dict(m) for m in ctx.messages]+ first = messages[0]+ messages[0] = {**first, "content": [{**first["content"][0], "cache_control": CC}]}+ ctx.messages = messages++ async def on_response(self, ctx: Any, response: Any, call_model: Any) -> None:+ return None+++def test_gateway_body_is_held_to_four_markers(headroom_client) -> None:+ register_turn_hook(_MarkerLeakHook())+ body = {+ "model": "claude-sonnet-4-5",+ "system": _system(2),+ "messages": _messages({4, 5}),+ "gateway": {"can_redrive": False, "can_relay_response": False},+ }+ resp = compress(headroom_client, body)+ assert resp.status_code == 200, resp.text+ data = resp.json()+ out = data["body"]+ assert (+ count_cache_breakpoints(out.get("system"), out["messages"], out.get("tools"))["total"] == 4+ )+ assert data["messages"] == out["messages"]+ assert "cache_breakpoint_budget" in data["transforms_applied"]diff --git a/tests/gateway/test_contract_response_half.py b/tests/gateway/test_contract_response_half.pyindex 5d5859c5d..597a0265f 100644--- a/tests/gateway/test_contract_response_half.py+++ b/tests/gateway/test_contract_response_half.py@@ -29,7 +29,15 @@ from tests.gateway.conftest import compress, count_loop_tasks, registry_offrom tests.gateway.samples import big_tool_historyRESPONSE_HALF = "/v1/compress/response"-DONE_KEYS = {"action", "turn_id", "response", "frozen_message_count", "usage_applied", "rounds"}+DONE_KEYS = {+ "action",+ "turn_id",+ "response",+ "frozen_message_count",+ "usage_applied",+ "rounds",+ "billed_usage",+}def _open_turn(client, session_id: str | None = "resp-s", **gateway: Any) -> dict[str, Any]:@@ -152,6 +160,7 @@ def test_done_with_null_response_and_turn_removed(headroom_client) -> None:assert data["response"] is Noneassert data["rounds"] == 0assert data["usage_applied"] is False # no cache signal in that usage+ assert data["billed_usage"] == {"input_tokens": 100, "output_tokens": 7}assert isinstance(data["frozen_message_count"], int)assert registry_of(headroom_client).get(turn["turn_id"]) is Noneagain = _post(headroom_client, {"turn_id": turn["turn_id"]})diff --git a/tests/gateway/test_redrive_loop.py b/tests/gateway/test_redrive_loop.pyindex f4e5a94c4..d242dcd6a 100644--- a/tests/gateway/test_redrive_loop.py+++ b/tests/gateway/test_redrive_loop.py@@ -123,11 +123,23 @@ def test_one_redrive_round_end_to_end(headroom_client, fake_provider, make_gatewdone = result.doneassert done is not Noneassert done["rounds"] == 1- # The hook's final answer IS the last provider response the gateway posted,- # so ``response`` may be null ("forward what you already have") or that same- # body; the client-visible invariant (asserted above) is what matters.- assert done["response"] is None or done["response"]["id"] == "chatcmpl-final"+ # The hook hands back the latest provider response object itself (no+ # replacement). After a re-drive Headroom still returns that response, with+ # usage summed over both provider calls, so the client never sees the last+ # round's usage alone.+ assert done["response"] is not None+ assert done["response"]["id"] == "chatcmpl-final"assert done["usage_applied"] is True # cached_tokens=300 on the final answer+ # The turn's bill covers both provider calls, not just the last one.+ assert done["billed_usage"]["input_tokens"] == 300 + 420+ assert done["billed_usage"]["output_tokens"] == 8 + 6+ assert done["billed_usage"]["cache_read_input_tokens"] == 300+ assert done["response"]["usage"]["prompt_tokens"] == 720+ assert done["response"]["usage"]["completion_tokens"] == 14+ assert done["response"]["usage"]["prompt_tokens_details"]["cached_tokens"] == 300+ # What the gateway hands the client is that cumulative response.+ assert result.final_response["usage"]["prompt_tokens"] == 720+ assert result.final_response["usage"]["completion_tokens"] == 14assert hook.rounds_driven == 1assert hook.queries == ["create issue"]# L-CLEAN@@ -221,7 +233,11 @@ def test_registry_max_redrives_env_caps_the_loop(# ``rounds`` past the cap and is answered ``done`` instead of ``redrive``.assert len(fake_provider.calls) == 3assert result.done["rounds"] == 3- assert result.done["response"] is None+ # The capped turn still ran three provider calls for one client call: the+ # latest provider answer comes back with usage summed over all three.+ assert result.done["response"]["choices"][0]["message"]["tool_calls"][0]["id"] == "s2"+ assert result.done["response"]["usage"]["prompt_tokens"] == 3 * 10+ assert result.done["response"]["usage"]["completion_tokens"] == 3 * 5assert registry_of(client).get(result.turn_id) is None@@ -258,6 +274,25 @@ def test_missing_response_on_redrive_turn_is_400(headroom_client) -> None:assert registry_of(headroom_client).get(data["turn_id"]) is not None+def test_failed_call_closes_a_redrive_turn_without_a_response(headroom_client) -> None:+ """L-GUARDS: a provider failure has nothing to re-drive. A gateway that+ only knows the status (LiteLLM's failure hooks see an exception, not a+ body) must still be able to close the turn and leave a ledger row."""+ redrive_hook_ext.register()+ data = headroom_client.post(+ "/v1/compress",+ json={**_chat_body(), "gateway": {"can_redrive": True}, "config": {"session_id": "fail"}},+ ).json()+ assert data["obligations"] == ["redrive"]+ resp = headroom_client.post(+ "/v1/compress/response", json={"turn_id": data["turn_id"], "round": 0, "status": 529}+ )+ assert resp.status_code == 200, resp.text+ assert resp.json()["action"] == "done"+ assert resp.json()["response"] is None+ assert registry_of(headroom_client).get(data["turn_id"]) is None++def test_non_200_provider_status_skips_hooks(headroom_client, fake_provider, make_gateway) -> None:"""L-GUARDS: a provider error is relayed, not re-driven; the turn completes."""from tests.gateway.fake_provider import ScriptedResponse@@ -449,3 +484,28 @@ def test_stale_retry_replays_last_redrive_without_double_billing(assert result.path == "loop"assert [c.request["round"] for c in result.response_half_calls] == [0, 1]assert result.final_response["choices"][0]["message"]["content"] == "final answer"+++def test_billed_usage_is_written_in_the_responses_own_shape() -> None:+ from headroom.proxy.gateway_turn import NormalizedUsage, _usage_in_shape++ billed = NormalizedUsage(input_tokens=720, output_tokens=14, cache_read=300, cache_write=50)+ anthropic = _usage_in_shape(+ {"input_tokens": 420, "output_tokens": 6, "service_tier": "standard"}, billed+ )+ assert anthropic == {+ "input_tokens": 720,+ "output_tokens": 14,+ "cache_read_input_tokens": 300,+ "cache_creation_input_tokens": 50,+ "service_tier": "standard",+ }+ chat = _usage_in_shape(+ {"prompt_tokens": 420, "completion_tokens": 6, "total_tokens": 426}, billed+ )+ assert chat == {+ "prompt_tokens": 720,+ "completion_tokens": 14,+ "total_tokens": 734,+ "prompt_tokens_details": {"cached_tokens": 300},+ }diff --git a/tests/test_compress_session_mode.py b/tests/test_compress_session_mode.pyindex cc3a23563..04a2c32d8 100644--- a/tests/test_compress_session_mode.py+++ b/tests/test_compress_session_mode.py@@ -566,3 +566,77 @@ def test_usage_404_for_ttl_expired_tracker() -> None:)assert resp.status_code == 404assert resp.json()["error"]["type"] == "unknown_session"+++# --------------------------------------------------------------------------- #+# cache_control markers stay bounded across replayed turns. #+# --------------------------------------------------------------------------- #+++def _count_message_markers(messages: list[dict]) -> int:+ total = 0+ for msg in messages:+ content = msg.get("content")+ if isinstance(content, list):+ total += sum(1 for b in content if isinstance(b, dict) and "cache_control" in b)+ return total+++def _anthropic_turns(n: int) -> list[dict]:+ """An agent loop in Anthropic shape, as a caller sends it on turn n: the+ breakpoint rides on the newest block only, moved forward every turn."""+ messages: list[dict] = [{"role": "user", "content": [{"type": "text", "text": "fix the bug"}]}]+ for i in range(n):+ messages.append(+ {+ "role": "assistant",+ "content": [+ {+ "type": "tool_use",+ "id": f"t{i}",+ "name": "bash",+ "input": {"cmd": f"step {i}"},+ }+ ],+ }+ )+ messages.append(+ {+ "role": "user",+ "content": [+ {+ "type": "tool_result",+ "tool_use_id": f"t{i}",+ # Large and compressible, so the fresh turn rewrites it+ # and the replay has compressed bytes to carry forward.+ "content": _big_tool_history()[2]["content"],+ }+ ],+ }+ )+ messages[-1]["content"][-1]["cache_control"] = {"type": "ephemeral"}+ return messages+++def test_session_turns_do_not_accumulate_cache_control_markers() -> None:+ # Anthropic rejects a request with more than four cache_control blocks.+ # Across replayed turns the returned markers must track the caller's+ # moving breakpoint, never accumulate behind it.+ with _make_client() as client:+ saved = 0+ for turn in range(1, 9):+ resp = client.post(+ "/v1/compress",+ json={+ "model": "claude-sonnet-5",+ "messages": _anthropic_turns(turn),+ "config": {"session_id": "markers-bounded"},+ },+ )+ assert resp.status_code == 200, resp.text+ body = resp.json()+ returned = body["messages"]+ saved += body["tokens_saved"]+ assert _count_message_markers(returned) == 1, (turn, _count_message_markers(returned))+ # Precondition: compression rewrote tool results, so replay was live.+ assert saved > 0diff --git a/tests/test_proxy/test_kompress_background_warmup.py b/tests/test_proxy/test_kompress_background_warmup.pynew file mode 100644index 000000000..69d68da0b--- /dev/null+++ b/tests/test_proxy/test_kompress_background_warmup.py@@ -0,0 +1,76 @@+"""Kompress loads on a background thread after startup, not inside a request.++Startup cannot build the model (native init before the port binds segfaults on+RHEL/CentOS 7-family hosts, #1908), so the first request that needed it paid the+whole load: a gateway benchmark turn stalled 21 s. The warm-up moves that cost+off the request path, and stays off on the affected glibc family.+"""++from __future__ import annotations++import threading+from types import SimpleNamespace++import pytest++from headroom.proxy.server import HeadroomProxy+++class _Kompress:+ def __init__(self) -> None:+ self.preloaded = threading.Event()++ def preload(self, *, allow_download: bool = True) -> str:+ self.preloaded.set()+ return "onnx"+++def _proxy_with(compressor: object | None) -> SimpleNamespace:+ router = SimpleNamespace(_get_kompress=lambda: compressor)+ pipeline = SimpleNamespace(transforms=[router])+ return SimpleNamespace(anthropic_pipeline=pipeline, openai_pipeline=pipeline)++[email protected](autouse=True)+def _no_delay(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("HEADROOM_KOMPRESS_WARMUP_DELAY_SECONDS", "0")+ monkeypatch.delenv("HEADROOM_KOMPRESS_WARMUP", raising=False)+++def test_warms_on_a_background_thread(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr("platform.libc_ver", lambda: ("glibc", "2.35"))+ monkeypatch.delenv("PYTEST_CURRENT_TEST", raising=False)+ kompress = _Kompress()+ assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(kompress)) is True+ assert kompress.preloaded.wait(5)+++def test_skipped_on_the_glibc_family_that_crashes(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr("platform.libc_ver", lambda: ("glibc", "2.17"))+ monkeypatch.delenv("PYTEST_CURRENT_TEST", raising=False)+ kompress = _Kompress()+ assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(kompress)) is False+ assert not kompress.preloaded.wait(0.2)+++def test_env_switch_wins_both_ways(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr("platform.libc_ver", lambda: ("glibc", "2.17"))+ monkeypatch.setenv("HEADROOM_KOMPRESS_WARMUP", "1")+ forced = _Kompress()+ assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(forced)) is True+ assert forced.preloaded.wait(5)++ monkeypatch.setattr("platform.libc_ver", lambda: ("glibc", "2.35"))+ monkeypatch.setenv("HEADROOM_KOMPRESS_WARMUP", "0")+ off = _Kompress()+ assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(off)) is False+++def test_no_kompress_installed_means_no_thread() -> None:+ assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(None)) is False+++def test_test_processes_do_not_warm_unless_forced(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr("platform.libc_ver", lambda: ("glibc", "2.35"))+ monkeypatch.setenv("PYTEST_CURRENT_TEST", "x")+ assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(_Kompress())) is Falsediff --git a/tests/test_transforms_content_router.py b/tests/test_transforms_content_router.pyindex c8994cb0b..7b0211c64 100644--- a/tests/test_transforms_content_router.py+++ b/tests/test_transforms_content_router.py@@ -1991,3 +1991,163 @@ def test_datetime_prefixed_user_prompt_survives_router() -> None:result = ContentRouter().compress(prompt)assert result.strategy_used is not CompressionStrategy.SEARCHassert "Please update the PR desc" in result.compressed+++# --- The caller's prompt stays verbatim on replaying paths -----------------+#+# The proxy's coding profile turns user-message compression on so tool+# observations inside user messages shrink. On a replaying path (proxy+# handlers, the /v1/compress session turn) that used to reach the prompt text+# too, and only a cache_control marker stopped it: Claude Code sets one, a+# plain agent does not, and its task statement went out with stop words+# stripped and pytest node ids mangled.++_TASK = "Fix this bug in the library source. " * 60+++def _prompt_router(monkeypatch: pytest.MonkeyPatch) -> tuple[ContentRouter, list[str]]:+ return _fresh_cc_router(monkeypatch)+++def test_opening_task_text_block_is_verbatim_under_replay(monkeypatch: pytest.MonkeyPatch) -> None:+ router, _ = _prompt_router(monkeypatch)+ messages = [{"role": "user", "content": [{"type": "text", "text": _TASK}]}]+ out = router.apply(+ messages,+ _ChurnTokenizer(),+ model_limit=100_000,+ prefix_replay_guaranteed=True,+ compress_user_messages=True,+ min_tokens_to_compress=1,+ )+ assert out.messages[0]["content"][0]["text"] == _TASK+++def test_opening_task_string_is_verbatim_under_replay(monkeypatch: pytest.MonkeyPatch) -> None:+ router, _ = _prompt_router(monkeypatch)+ messages = [{"role": "user", "content": _TASK}]+ out = router.apply(+ messages,+ _ChurnTokenizer(),+ model_limit=100_000,+ prefix_replay_guaranteed=True,+ compress_user_messages=True,+ min_tokens_to_compress=1,+ )+ assert out.messages[0]["content"] == _TASK+++def test_newest_user_turn_keeps_text_but_compresses_tool_results(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ router, calls = _prompt_router(monkeypatch)+ follow_up = "Now also handle the async path. " * 60+ messages = [+ {"role": "user", "content": "run the tests"},+ {+ "role": "assistant",+ "content": [{"type": "tool_use", "id": "t1", "name": "bash", "input": {}}],+ },+ {+ "role": "user",+ "content": [+ {"type": "tool_result", "tool_use_id": "t1", "content": _FRESH_CC_OUTPUT},+ {"type": "text", "text": follow_up},+ ],+ },+ ]+ out = router.apply(+ messages,+ _ChurnTokenizer(),+ model_limit=100_000,+ prefix_replay_guaranteed=True,+ compress_user_messages=True,+ min_tokens_to_compress=1,+ )+ blocks = out.messages[2]["content"]+ assert blocks[0]["content"].endswith("[compressed]")+ assert blocks[1]["text"] == follow_up+ assert follow_up not in calls+++def test_text_harness_observation_string_still_compresses_under_replay(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ # A text harness returns its tool output as a role:user string after the+ # assistant turn. That is an observation, not the prompt.+ router, calls = _prompt_router(monkeypatch)+ messages = [+ {"role": "user", "content": "run the tests"},+ {"role": "assistant", "content": "```bash\npytest -q\n```"},+ {"role": "user", "content": _FRESH_CC_OUTPUT},+ ]+ router.apply(+ messages,+ _ChurnTokenizer(),+ model_limit=100_000,+ prefix_replay_guaranteed=True,+ compress_user_messages=True,+ min_tokens_to_compress=1,+ )+ assert _FRESH_CC_OUTPUT in calls+++def test_follow_up_prompt_string_after_assistant_turn_is_verbatim_under_replay(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ # A plain-string follow-up after an assistant reply is the caller's new+ # instruction, the same as the list-content case above.+ router, calls = _prompt_router(monkeypatch)+ follow_up = "Now also handle the async path in tenacity/asyncio. " * 60+ messages = [+ {"role": "user", "content": "run the tests"},+ {"role": "assistant", "content": "All 167 tests pass on the sync path."},+ {"role": "user", "content": follow_up},+ ]+ out = router.apply(+ messages,+ _ChurnTokenizer(),+ model_limit=100_000,+ prefix_replay_guaranteed=True,+ compress_user_messages=True,+ min_tokens_to_compress=1,+ )+ assert out.messages[2]["content"] == follow_up+ assert follow_up not in calls+++def test_text_harness_observation_text_block_still_compresses_under_replay(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ # The list-content twin of the string observation above: both shapes+ # decide "prompt or observation" the same way.+ router, calls = _prompt_router(monkeypatch)+ messages = [+ {"role": "user", "content": "run the tests"},+ {"role": "assistant", "content": "```bash\npytest -q\n```"},+ {"role": "user", "content": [{"type": "text", "text": _FRESH_CC_OUTPUT}]},+ ]+ router.apply(+ messages,+ _ChurnTokenizer(),+ model_limit=100_000,+ prefix_replay_guaranteed=True,+ compress_user_messages=True,+ min_tokens_to_compress=1,+ )+ assert _FRESH_CC_OUTPUT in calls+++def test_prompt_text_compresses_without_replay_guarantee(monkeypatch: pytest.MonkeyPatch) -> None:+ # SDK / document callers that opt into user compression keep it: a+ # spreadsheet or pasted document in a user message is the payload.+ router, calls = _prompt_router(monkeypatch)+ messages = [{"role": "user", "content": [{"type": "text", "text": _TASK}]}]+ router.apply(+ messages,+ _ChurnTokenizer(),+ model_limit=100_000,+ compress_user_messages=True,+ min_tokens_to_compress=1,+ )+ assert _TASK in callsdiff --git a/wiki/proxy.md b/wiki/proxy.mdindex 584f4b239..69d3f558c 100644--- a/wiki/proxy.md+++ b/wiki/proxy.md@@ -348,7 +348,7 @@ Optionally relay the provider's usage for attribution with `POST /v1/usage` `{"s**Gateway mode (two-half turn contract).** Add a top-level `gateway` object (`{}` is enough) and one model turn becomes two calls, so a gateway that never lets Headroom see the provider response can still run transforms whose reload step needs it — the proxy's "no shrink without reload" rule at the API boundary. `gateway` fields: `can_redrive` (default `false`: the gateway can call the provider again with a request Headroom hands it; send `false` when streaming), `can_relay_response` (default `false`: the gateway will post status/usage after each turn), `session_affinity` (default `true`; `false` disables re-driving because pending turns are in-process), `plugin_version` (diagnostics). Every other top-level field (`system`, `tools`, `temperature`, …) passes through; `tools` is compacted deterministically (`tool_schema_compaction`; `tool_desc_compaction` with `HEADROOM_TOOL_DESC_MAX_CHARS`); extension turn hooks run, stream-safe-only unless `can_redrive` and `session_affinity` both hold. The response adds `body` (the complete provider request — forward it as-is; never contains `config`, `gateway`, `token_budget`), `turn_id`, `route` (`{model, provider, service_tier, reason}`, advisory; a routing extension's model is also written into `body.model`), `obligations` (`redrive` and/or `relay_usage`) and a `gateway` echo. Fail-open answers carry the same keys with originals and `obligations: []`. A turn is registered only when `obligations` is non-empty; with `relay_usage` the `/stats` record waits for the response half. Knobs: `HEADROOM_GATEWAY_TURN_TTL_SECONDS` (120), `HEADROOM_GATEWAY_MAX_PENDING_TURNS` (10000), `HEADROOM_GATEWAY_MAX_REDRIVES` (8). With `config.mode: "ccr"` and re-drive allowed, `headroom_retrieve` is injected into `body.tools` (`ccr_tool_injected`) and the response half answers the model's retrieval calls itself.-`POST /v1/compress/response` (same exposure rules as `/v1/compress`) is the response half: `{"turn_id", "status": 200, "latency_ms", "usage": {...}, "response": {...}}`. `usage` accepts Anthropic, OpenAI chat (`prompt_tokens_details.cached_tokens`), OpenAI Responses (`input_tokens_details.cached_tokens`) and Kong's flat `cached_tokens` shapes, or the whole provider body (a nested `usage` key is descended into once); billed counters are summed across re-drive rounds. `response` is required when the turn carries `redrive`. Answers: `{"action": "done", "turn_id", "response": <replacement or null — forward what you hold>, "frozen_message_count", "usage_applied", "rounds"}` or `{"action": "redrive", "turn_id", "request": <full provider body to send>, "round"}` — post the provider's JSON back under the same `turn_id`; past `HEADROOM_GATEWAY_MAX_REDRIVES` the turn ends with `done` and `response: null`. Errors: 400 `invalid_request` / `missing_response`, 404 `unknown_turn` (unregistered, finished, or expired), 409 `turn_busy`.+`POST /v1/compress/response` (same exposure rules as `/v1/compress`) is the response half: `{"turn_id", "status": 200, "latency_ms", "usage": {...}, "response": {...}}`. `usage` accepts Anthropic, OpenAI chat (`prompt_tokens_details.cached_tokens`), OpenAI Responses (`input_tokens_details.cached_tokens`) and Kong's flat `cached_tokens` shapes, or the whole provider body (a nested `usage` key is descended into once); billed counters are summed across re-drive rounds. `response` is required when the turn carries `redrive` and the provider call succeeded; a failed call closes the turn with its `status` alone. Answers: `{"action": "done", "turn_id", "response": <replacement, the latest provider response after a re-drive, or null — forward what you hold>, "frozen_message_count", "usage_applied", "rounds", "billed_usage"}` or `{"action": "redrive", "turn_id", "request": <full provider body to send>, "round"}` — post the provider's JSON back under the same `turn_id`; `billed_usage` sums every counter (cache reads and writes included) across rounds in Anthropic keys, and a re-driven turn's `response` is never null and reports that total as its `usage`; past `HEADROOM_GATEWAY_MAX_REDRIVES` the turn ends with `done` and the latest provider response. Errors: 400 `invalid_request` / `missing_response`, 404 `unknown_turn` (unregistered, finished, or expired), 409 `turn_busy`.**Kong plugin.** [kong-plugin-headroom](https://github.com/headroomlabs-ai/kong-plugin-headroom) (its own repo; `luarocks install kong-plugin-headroom`) implements both halves for Kong Gateway 3.9 (session id from a header, usage relay from the `log` phase, re-drive loop in `access`). The contract itself is installed in Headroom through the compress-turn seam (`headroom.proxy.compress_turn`, `HEADROOM_GATEWAY_CONTRACT`), the same seam a third-party contract would use.