Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(proxy): gateway turn fixes found by benchmarking behind a gateway (#3619)

Tejas Chopraaceff2ea9dbd
diff --git a/docs/content/docs/proxy.mdx b/docs/content/docs/proxy.mdx
index d077a6ce8..8e2a0ee27 100644
--- a/docs/content/docs/proxy.mdx
+++ b/docs/content/docs/proxy.mdx
@@ -685,7 +685,7 @@ The response half. Same exposure rules as `/v1/compress`: loopback-only unless `
| `latency_ms` | number | no | Gateway-measured provider latency, added to the turn's recorded latency. Must be non-negative. |
| `round` | integer | no | Which provider call this response answers: `0` for the original forward, `n` for the `n`-th re-drive (the `round` this endpoint returned). Makes the call idempotent: a retried relay for a round already answered gets the same `redrive` answer again instead of being fed to the hook, and its usage is not billed twice. A round ahead of the turn is a `400`. |
| `usage` | object | no | The provider's usage block, in any accepted shape (below). |
-| `response` | object | no | The full provider response JSON. **Required** when the turn carries `redrive`. |
+| `response` | object | no | The full provider response JSON. **Required** when the turn carries `redrive` and the provider call succeeded (`2xx`). A failed call closes the turn with its `status` alone. |
| `model` | string | no | Provider-reported model. Accepted, not read today. |
Accepted `usage` shapes — the whole provider response body works too, since a nested `usage` key is descended into once:
@@ -706,11 +706,12 @@ There are two answers. `done` ends the turn:
"response": null,
"frozen_message_count": 12,
"usage_applied": true,
- "rounds": 0
+ "rounds": 0,
+ "billed_usage": { "input_tokens": 3120, "output_tokens": 212, "cache_read_input_tokens": 20414 }
}
```
-`response` is a replacement to send to the client when a hook produced one; otherwise it is `null` and you forward the provider response you already hold. `frozen_message_count` is the session's count after the usage was applied (`null` for a stateless turn). `usage_applied` is `false` when the usage carried no cache signal, the session had expired, or its lock was busy — never an error, since the turn is finishing and there is nothing to retry with. After `done` the turn is gone.
+`response` is what to send to the client. On a turn with no re-drive it is the hook's replacement when there is one, otherwise `null` and you forward the provider response you already hold. After one or more re-drives it is never `null`: it is the hook's replacement or the latest provider response, with `usage` set to the whole turn's bill. `frozen_message_count` is the session's count after the usage was applied (`null` for a stateless turn). `usage_applied` is `false` when the usage carried no cache signal, the session had expired, or its lock was busy — never an error, since the turn is finishing and there is nothing to retry with. `billed_usage` is what the provider billed for the whole turn: every counter, cache reads and writes included, summed across all rounds, in Anthropic keys (`null` when no usage was relayed). That total is the re-driven `response`'s `usage`, since the client made one call. After `done` the turn is gone.
`redrive` asks for another provider call:
@@ -723,7 +724,7 @@ There are two answers. `done` ends the turn:
}
```
-Send `request` to the provider exactly as given — it is the provider body with the hook's messages and possibly reloaded tools — and post the provider's JSON back under the same `turn_id`. Past `HEADROOM_GATEWAY_MAX_REDRIVES` the turn ends with `done` and `response: null`. Hook failures never fail the turn either: the answer is `done` with `response: null` and the gateway forwards what it holds.
+Send `request` to the provider exactly as given — it is the provider body with the hook's messages and possibly reloaded tools — and post the provider's JSON back under the same `turn_id`. Past `HEADROOM_GATEWAY_MAX_REDRIVES` the turn ends with `done` and the latest provider response. Hook failures never fail the turn either: the answer is `done`, with the latest provider response after a re-drive, or `response: null` before one, and the gateway forwards what it gets.
| Status | Body | When |
| --- | --- | --- |
diff --git a/headroom/proxy/gateway_extension.py b/headroom/proxy/gateway_extension.py
index 6eef84ca8..1bbac9cd9 100644
--- a/headroom/proxy/gateway_extension.py
+++ b/headroom/proxy/gateway_extension.py
@@ -187,6 +187,27 @@ class GatewayTurn:
input_tokens=tokens_before,
transforms=transforms,
)
+ if self.provider == "anthropic":
+ # Last stop before the body leaves: a request over Anthropic's
+ # cache_control budget is a guaranteed 400 on the provider.
+ from headroom.proxy.helpers import enforce_cache_breakpoint_budget
+
+ provider_body = self._fields["body"]
+ client_messages = self.body.get("messages") if isinstance(self.body, dict) else None
+ system, guarded, tools, budget = enforce_cache_breakpoint_budget(
+ provider_body.get("system"),
+ provider_body.get("messages"),
+ provider_body.get("tools"),
+ client_messages=client_messages,
+ request_id=self._turn_id or "",
+ )
+ if budget["repaired"]:
+ if provider_body.get("system") is not None:
+ provider_body["system"] = system
+ provider_body["messages"] = guarded
+ if provider_body.get("tools") is not None:
+ provider_body["tools"] = tools
+ transforms.append("cache_breakpoint_budget")
# I-BODY: top-level `messages` and `body.messages` are one list.
return FinishedTurn(
fields=self._fields,
diff --git a/headroom/proxy/gateway_turn.py b/headroom/proxy/gateway_turn.py
index b1899f5c4..851a03cae 100644
--- a/headroom/proxy/gateway_turn.py
+++ b/headroom/proxy/gateway_turn.py
@@ -263,6 +263,37 @@ class NormalizedUsage:
cache_write=other.cache_write if other.has_cache_signal else self.cache_write,
)
+ def summed_with(self, other: NormalizedUsage) -> NormalizedUsage:
+ """Sum every counter across re-drive rounds: what the provider billed
+ for the whole turn, cache reads and writes of every round included.
+ ``merged_with`` keeps the latest cache state for freeze decisions;
+ this is the bill."""
+
+ def _sum(a: int | None, b: int | None) -> int | None:
+ if a is None and b is None:
+ return None
+ return (a or 0) + (b or 0)
+
+ return NormalizedUsage(
+ input_tokens=_sum(self.input_tokens, other.input_tokens),
+ output_tokens=_sum(self.output_tokens, other.output_tokens),
+ cache_read=_sum(self.cache_read, other.cache_read),
+ cache_write=_sum(self.cache_write, other.cache_write),
+ )
+
+ def as_anthropic(self) -> dict[str, int]:
+ """Anthropic ``usage`` keys, only for the counters that were reported."""
+ out: dict[str, int] = {}
+ if self.input_tokens is not None:
+ out["input_tokens"] = self.input_tokens
+ if self.output_tokens is not None:
+ out["output_tokens"] = self.output_tokens
+ if self.cache_read is not None:
+ out["cache_read_input_tokens"] = self.cache_read
+ if self.cache_write is not None:
+ out["cache_creation_input_tokens"] = self.cache_write
+ return out
+
def _usage_int(usage: dict[str, Any], name: str) -> int | None:
if name not in usage:
@@ -475,8 +506,11 @@ class PendingTurn:
runner: SuspendedHookRunner | None = None
tags: dict[str, Any] = field(default_factory=dict)
client: str | None = None
- # Billed usage accumulated across every response-half call for the turn.
+ # Usage accumulated across every response-half call for the turn: input
+ # and output summed, cache fields from the latest round (freeze state).
usage: NormalizedUsage | None = None
+ # Every counter summed across rounds: the turn's bill.
+ billed: NormalizedUsage | None = None
# A second response-half call while one is being driven would race the
# parked coroutine; the handler answers 409 instead.
in_flight: bool = False
@@ -1325,6 +1359,31 @@ def _redrive_payload(turn: PendingTurn, step: Step) -> dict[str, Any]:
}
+def _usage_in_shape(existing: dict[str, Any], billed: NormalizedUsage) -> dict[str, Any]:
+ """Overwrite ``existing`` usage counters with ``billed``, keeping its shape.
+
+ An OpenAI chat usage block (``prompt_tokens``) gets chat keys; anything
+ else gets Anthropic keys. Counters the provider never reported stay out.
+ """
+ out = dict(existing)
+ if "prompt_tokens" in existing or "completion_tokens" in existing:
+ if billed.input_tokens is not None:
+ out["prompt_tokens"] = billed.input_tokens
+ if billed.output_tokens is not None:
+ out["completion_tokens"] = billed.output_tokens
+ if billed.input_tokens is not None or billed.output_tokens is not None:
+ out["total_tokens"] = (billed.input_tokens or 0) + (billed.output_tokens or 0)
+ if billed.cache_read is not None:
+ details = existing.get("prompt_tokens_details")
+ out["prompt_tokens_details"] = {
+ **(details if isinstance(details, dict) else {}),
+ "cached_tokens": billed.cache_read,
+ }
+ return out
+ out.update(billed.as_anthropic())
+ return out
+
+
async def handle_compress_response(proxy: Any, request: Any) -> Any:
"""``POST /v1/compress/response`` — finish or re-drive a pending turn."""
from fastapi.responses import JSONResponse
@@ -1375,7 +1434,9 @@ async def handle_compress_response(proxy: Any, request: Any) -> Any:
"unknown_turn",
f"No pending turn {turn_id!r} (never registered, already finished, or expired).",
)
- if OBLIGATION_REDRIVE in turn.obligations and response is None:
+ # A failed provider call has nothing to re-drive: its status (and usage,
+ # if any) closes the turn. Only a successful response must carry the body.
+ if OBLIGATION_REDRIVE in turn.obligations and response is None and 200 <= status < 300:
return _error(
400,
"missing_response",
@@ -1416,6 +1477,7 @@ async def handle_compress_response(proxy: Any, request: Any) -> Any:
try:
if usage is not None:
turn.usage = usage if turn.usage is None else turn.usage.merged_with(usage)
+ turn.billed = usage if turn.billed is None else turn.billed.summed_with(usage)
step: Step | None = None
if turn.hooks_armed and turn.ctx is not None:
@@ -1461,7 +1523,10 @@ async def handle_compress_response(proxy: Any, request: Any) -> Any:
frozen, applied = await apply_session_usage(proxy, turn, turn.usage)
if turn.outcome_draft is not None:
outcome = complete_outcome(
- turn.outcome_draft, turn.usage, status=status, latency_ms=latency_ms
+ turn.outcome_draft,
+ turn.billed or turn.usage,
+ status=status,
+ latency_ms=latency_ms,
)
turn.outcome_draft = None
await proxy._record_request_outcome(outcome)
@@ -1470,6 +1535,25 @@ async def handle_compress_response(proxy: Any, request: Any) -> Any:
)
if registry is not None:
registry.pop(turn_id)
+ billed = turn.billed.as_anthropic() if turn.billed is not None else None
+ # After a re-drive the answer is either the hook's replacement or, when
+ # the hook handed back the latest provider response (``response: null``),
+ # the response the gateway posted on this call. Either way the client made
+ # one call, so its response reports what the provider billed for all of
+ # it, the way a server-side tool loop does, in the response's own usage
+ # shape. A turn with no re-drive keeps ``response: null``: the gateway's
+ # held response already carries the whole bill.
+ answer = final_response if final_response is not None else response
+ if (
+ turn.rounds > 0
+ and billed
+ and isinstance(answer, dict)
+ and isinstance(answer.get("usage"), dict)
+ ):
+ final_response = {
+ **answer,
+ "usage": _usage_in_shape(answer["usage"], turn.billed),
+ }
return JSONResponse(
{
"action": "done",
@@ -1478,6 +1562,8 @@ async def handle_compress_response(proxy: Any, request: Any) -> Any:
"frozen_message_count": frozen,
"usage_applied": applied,
"rounds": turn.rounds,
+ # Every counter summed across rounds, Anthropic keys.
+ "billed_usage": billed,
}
)
finally:
diff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.py
index b6e8cf36e..cb7078e54 100644
--- a/headroom/proxy/handlers/openai.py
+++ b/headroom/proxy/handlers/openai.py
@@ -10113,7 +10113,17 @@ class OpenAIHandlerMixin:
# the pipeline introduced — byte-identical is the contract
# the caller forwards on.
turn = finalize_turn(result.messages, messages, prev_original, prev_returned)
- final = turn.messages
+ # A replay can carry an earlier turn's cache_control marker
+ # after the caller moved its breakpoint forward, and
+ # Anthropic rejects more than four. Keep message markers at
+ # the caller's current positions, as the /v1/messages
+ # handler does after its own replay. The gateway's finish
+ # step still enforces the hard budget on the whole body.
+ from headroom.cache.prefix_tracker import normalize_message_cache_control
+
+ final = normalize_message_cache_control(
+ turn.messages, prev_returned, client_messages=messages
+ )
if _turn is not None:
# AFTER the replay overlay, BEFORE update_from_result /
# record_returned: the turn's output is what the caller
diff --git a/headroom/proxy/helpers.py b/headroom/proxy/helpers.py
index 2d4f1d700..8a33186fe 100644
--- a/headroom/proxy/helpers.py
+++ b/headroom/proxy/helpers.py
@@ -740,6 +740,129 @@ def enforce_cache_control_ttl_order(
return system, messages, tools, stats
+#: Anthropic rejects a request with more than this many cache_control blocks
+#: across tools, system and messages ("A maximum of 4 blocks with
+#: cache_control may be provided").
+ANTHROPIC_MAX_CACHE_BREAKPOINTS = 4
+
+
+def enforce_cache_breakpoint_budget(
+ system: Any,
+ messages: Any,
+ tools: Any,
+ *,
+ client_messages: Any = None,
+ limit: int = ANTHROPIC_MAX_CACHE_BREAKPOINTS,
+ request_id: str = "",
+) -> tuple[Any, Any, Any, dict[str, Any]]:
+ """Last-stop guard: never forward more ``cache_control`` blocks than allowed.
+
+ A request over the limit is a guaranteed 400, and the client cannot have
+ caused it with its own markers, so whatever pushed it over was added on
+ the way through. Repairs in the order that costs the least caching:
+
+ 1. Re-place message markers at the client's own positions
+ (:func:`~headroom.cache.prefix_tracker.normalize_message_cache_control`),
+ which drops markers a replay or transform carried in.
+ 2. If still over, drop the OLDEST message markers first: the newest one is
+ the client's write anchor for the growing tail, and system/tools
+ markers cover the hottest, longest-lived prefix.
+ 3. If still over, drop system markers, then tools markers, earliest first.
+
+ Returns ``(system, messages, tools, stats)``; the inputs come back by
+ identity when nothing needed repairing. Logs one WARNING per repair with
+ the per-section counts so the source of the extra marker is traceable.
+ """
+ before = count_cache_breakpoints(system, messages, tools)
+ stats: dict[str, Any] = {"repaired": False, "before": before, "after": before}
+ if before["total"] <= limit:
+ return system, messages, tools, stats
+
+ if isinstance(messages, list) and client_messages is not None:
+ from headroom.cache.prefix_tracker import normalize_message_cache_control
+
+ messages = normalize_message_cache_control(messages, None, client_messages=client_messages)
+
+ for section in ("messages", "system", "tools"):
+ excess = count_cache_breakpoints(system, messages, tools)["total"] - limit
+ if excess <= 0:
+ break
+ budget = {"left": excess}
+ # Removal, oldest first in Anthropic's evaluation order. Copy-on-write:
+ # the forwarded body shares structure with the session snapshot.
+ if section == "messages" and isinstance(messages, list):
+ rebuilt: list[Any] = []
+ for msg in messages:
+ if budget["left"] <= 0 or not isinstance(msg, dict):
+ rebuilt.append(msg)
+ continue
+ content = msg.get("content")
+ if isinstance(content, list):
+ new_blocks = []
+ for block in content:
+ if budget["left"] > 0 and isinstance(block, dict):
+ inner = block.get("content")
+ if isinstance(inner, list):
+ new_inner = []
+ for sub in inner:
+ if (
+ budget["left"] > 0
+ and isinstance(sub, dict)
+ and "cache_control" in sub
+ ):
+ sub = {k: v for k, v in sub.items() if k != "cache_control"}
+ budget["left"] -= 1
+ new_inner.append(sub)
+ block = {**block, "content": new_inner}
+ if budget["left"] > 0 and "cache_control" in block:
+ block = {k: v for k, v in block.items() if k != "cache_control"}
+ budget["left"] -= 1
+ new_blocks.append(block)
+ msg = {**msg, "content": new_blocks}
+ if budget["left"] > 0 and "cache_control" in msg:
+ msg = {k: v for k, v in msg.items() if k != "cache_control"}
+ budget["left"] -= 1
+ rebuilt.append(msg)
+ messages = rebuilt
+ elif section in ("system", "tools"):
+ holders = system if section == "system" else tools
+ if isinstance(holders, list):
+ rebuilt_h = []
+ for holder in holders:
+ if (
+ budget["left"] > 0
+ and isinstance(holder, dict)
+ and "cache_control" in holder
+ ):
+ holder = {k: v for k, v in holder.items() if k != "cache_control"}
+ budget["left"] -= 1
+ rebuilt_h.append(holder)
+ if section == "system":
+ system = rebuilt_h
+ else:
+ tools = rebuilt_h
+
+ after = count_cache_breakpoints(system, messages, tools)
+ stats.update({"repaired": True, "after": after})
+ logger.warning(
+ "event=cache_breakpoint_budget request_id=%s limit=%d "
+ "before_total=%d before_system=%d before_tools=%d before_messages=%d "
+ "after_total=%d after_system=%d after_tools=%d after_messages=%d; "
+ "the outbound request carried more cache_control blocks than the provider accepts",
+ request_id,
+ limit,
+ before["total"],
+ before["system"],
+ before["tools"],
+ before["messages"],
+ after["total"],
+ after["system"],
+ after["tools"],
+ after["messages"],
+ )
+ return system, messages, tools, stats
+
+
def log_cache_breakpoints(
*,
request_id: str | None,
diff --git a/headroom/proxy/server.py b/headroom/proxy/server.py
index 9cd7694d7..1e690a972 100644
--- a/headroom/proxy/server.py
+++ b/headroom/proxy/server.py
@@ -1873,6 +1873,75 @@ class HeadroomProxy(
return eager_status, transform_statuses
+ def _start_kompress_background_warmup(self) -> bool:
+ """Load the Kompress model on a daemon thread once startup has returned.
+
+ Startup must not build the model (native init before the port binds
+ segfaults on RHEL/CentOS 7-family hosts, #1908), so it used to load
+ inside the first request that needed it: a benchmark turn through a
+ gateway stalled 21 s while transformers, torch and the ONNX session
+ came up. Loading on a background thread after a short delay moves that
+ cost off the request path. Skipped on glibc older than 2.28 (the
+ affected host family) and when ``HEADROOM_KOMPRESS_WARMUP`` is ``0``;
+ ``1`` forces it. Returns ``True`` when a warm-up thread was started.
+ """
+ raw = os.environ.get("HEADROOM_KOMPRESS_WARMUP", "").strip().lower()
+ if raw in ("0", "false", "no", "off"):
+ return False
+ if not raw and os.environ.get("PYTEST_CURRENT_TEST"):
+ # Test apps boot by the hundred; none of them should load a model.
+ return False
+ if raw not in ("1", "true", "yes", "on"):
+ import platform
+
+ libc, version = platform.libc_ver()
+ if libc == "glibc":
+ try:
+ major, minor = (int(part) for part in version.split(".")[:2])
+ except ValueError:
+ major, minor = 0, 0
+ if (major, minor) < (2, 28):
+ return False
+ compressor = None
+ for pipeline in (self.anthropic_pipeline, self.openai_pipeline):
+ for transform in getattr(pipeline, "transforms", []):
+ getter = getattr(transform, "_get_kompress", None)
+ if getter is None:
+ continue
+ try:
+ compressor = getter()
+ except Exception:
+ compressor = None
+ if compressor is not None and hasattr(compressor, "preload"):
+ break
+ compressor = None
+ if compressor is not None:
+ break
+ if compressor is None:
+ return False
+
+ try:
+ delay = float(os.environ.get("HEADROOM_KOMPRESS_WARMUP_DELAY_SECONDS", "2") or 2)
+ except ValueError:
+ delay = 2.0
+
+ def _warm() -> None:
+ time.sleep(max(0.0, delay))
+ started = time.monotonic()
+ try:
+ backend = compressor.preload(allow_download=True)
+ except Exception as exc: # the lazy request path still loads on first use
+ logger.warning("Kompress background warm-up failed: %s", exc)
+ return
+ logger.info(
+ "Kompress: warmed in the background in %.0f ms (backend %s)",
+ (time.monotonic() - started) * 1000,
+ backend,
+ )
+
+ threading.Thread(target=_warm, name="kompress-warmup", daemon=True).start()
+ return True
+
async def startup(self):
"""Initialize async resources."""
self._get_shutdown_event().clear()
@@ -2013,7 +2082,10 @@ class HeadroomProxy(
if self._kompress_status == "enabled":
logger.info("Kompress: ENABLED (ModernBERT token compressor)")
elif self._kompress_status == "deferred":
- logger.info("Kompress: DEFERRED (model loads on first request)")
+ if self._start_kompress_background_warmup():
+ logger.info("Kompress: DEFERRED (warming in the background after startup)")
+ else:
+ logger.info("Kompress: DEFERRED (model loads on first request)")
elif self.config.optimize:
logger.info("Kompress: not installed (pip install headroom-ai[ml] for ML compression)")
diff --git a/headroom/transforms/content_router.py b/headroom/transforms/content_router.py
index 9734db464..207650681 100644
--- a/headroom/transforms/content_router.py
+++ b/headroom/transforms/content_router.py
@@ -617,6 +617,23 @@ def _fenced_shell_command(content: Any) -> str:
return m.group(1).strip() if m else ""
+def _answers_fenced_command(messages: list[dict[str, Any]], index: int) -> bool:
+ """True when the user message at ``index`` replies to a fenced shell command.
+
+ Text-based harnesses (mini-swe-agent and similar) send a command in a fenced
+ block in the assistant turn and return its output as the next plain user
+ message. That message is a tool observation, not the caller's prompt. Walks
+ back to the nearest assistant turn, stopping at an earlier user turn.
+ """
+ for j in range(index - 1, -1, -1):
+ role = messages[j].get("role") if isinstance(messages[j], dict) else None
+ if role == "assistant":
+ return bool(_fenced_shell_command(messages[j].get("content")))
+ if role == "user":
+ return False
+ return False
+
+
_READ_VERBS = ("cat", "head", "tail", "nl", "bat", "less", "more")
# Machine-generated dependency lockfiles detect as PLAIN_TEXT (so the content-based
@@ -4973,6 +4990,16 @@ class ContentRouter(Transform):
# --- Adaptive parameters based on context pressure ---
num_messages = len(messages)
+ # The opening prompt: every user message before the first assistant
+ # turn. See `protect_prompt_text` in _process_content_blocks.
+ first_assistant_index = next(
+ (
+ idx
+ for idx, msg in enumerate(messages)
+ if isinstance(msg, dict) and msg.get("role") == "assistant"
+ ),
+ num_messages,
+ )
model_limit = kwargs.get("model_limit", 0)
# Adaptive Read protection: protect a fraction of recent messages
@@ -5177,6 +5204,23 @@ class ContentRouter(Transform):
bias = 1.0 # Default bias, may be overridden for tool messages
messages_from_end = num_messages - i
+ # The caller's own words stay verbatim on a replaying path even
+ # when user messages are compressible for their tool observations:
+ # the opening prompt (a task statement with test ids and paths)
+ # and the text of the newest user turn. Lossy text compression
+ # there rewrites what the model is asked to do, and nothing but a
+ # cache_control marker used to stop it -- a marker Claude Code sets
+ # and plain agents do not. The one newest user turn that is not a
+ # prompt is a text harness's tool observation: the reply to a
+ # fenced shell command in the assistant turn before it.
+ prompt_turn = (
+ prefix_replay_guaranteed
+ and role == "user"
+ and (
+ i < first_assistant_index
+ or (messages_from_end == 1 and not _answers_fenced_command(messages, i))
+ )
+ )
# Handle list content (Anthropic format with content blocks)
if isinstance(content, list):
@@ -5199,6 +5243,7 @@ class ContentRouter(Transform):
skip_system=skip_system,
compress_assistant_text_blocks=compress_assistant_text_blocks,
prefix_replay_guaranteed=prefix_replay_guaranteed,
+ protect_prompt_text=prompt_turn,
)
result_slots[i] = transformed_message
route_counts["content_blocks"] += 1
@@ -5315,8 +5360,9 @@ class ContentRouter(Transform):
route_counts["read_protected"] += 1
continue
- # Protection 1: Never compress user messages (unless overridden)
- if skip_user and role == "user":
+ # Protection 1: Never compress user messages (unless overridden),
+ # and never the caller's prompt on a replaying path.
+ if role == "user" and (skip_user or prompt_turn):
result_slots[i] = message
transforms_applied.append("router:protected:user_message")
route_counts["user_msg"] += 1
@@ -6089,6 +6135,7 @@ class ContentRouter(Transform):
skip_system: bool = True,
compress_assistant_text_blocks: bool = False,
prefix_replay_guaranteed: bool = False,
+ protect_prompt_text: bool = False,
) -> dict[str, Any]:
"""Process content blocks (Anthropic format) for compression.
@@ -6150,6 +6197,10 @@ class ContentRouter(Transform):
skip_system: If True, never compress text blocks in system-role messages.
compress_assistant_text_blocks: If True, allow compressing text blocks in
assistant-role messages. Default False (cache-safe).
+ protect_prompt_text: If True, text blocks in this user message are the
+ caller's prompt (the opening task or the newest user turn) and stay
+ verbatim even when ``skip_user`` is False; tool_result blocks in the
+ same message are still compressible.
Returns:
Transformed message with compressed content blocks.
@@ -6162,7 +6213,7 @@ class ContentRouter(Transform):
# outputs and compress freely; assistant defaults to skip (cache
# safety) with explicit opt-in; unknown roles default to skip.
if role == "user":
- protect_text_blocks = skip_user
+ protect_text_blocks = skip_user or protect_prompt_text
elif role in {"system", "developer"}:
protect_text_blocks = skip_system
elif role == "assistant":
diff --git a/tests/gateway/test_cache_breakpoint_budget.py b/tests/gateway/test_cache_breakpoint_budget.py
new file mode 100644
index 000000000..0c1b43aeb
--- /dev/null
+++ b/tests/gateway/test_cache_breakpoint_budget.py
@@ -0,0 +1,109 @@
+"""The gateway body never carries more cache_control blocks than Anthropic accepts.
+
+A LiteLLM benchmark run saw ``A maximum of 4 blocks with cache_control may be
+provided. Found 5.`` on a turn whose only Headroom work was tool-result
+compression and tool schema compaction. The client (Claude Code) never sends
+more than four, so the fifth was added on the way through. Whatever the path,
+the last stop before the body leaves must hold the budget.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+from headroom.proxy.helpers import count_cache_breakpoints, enforce_cache_breakpoint_budget
+from headroom.proxy.turn_hooks import register_turn_hook
+from tests.gateway.conftest import compress
+
+CC = {"type": "ephemeral"}
+
+
+def _system(markers: int) -> list[dict[str, Any]]:
+ return [
+ {
+ "type": "text",
+ "text": f"system part {i}",
+ **({"cache_control": CC} if i < markers else {}),
+ }
+ for i in range(2)
+ ]
+
+
+def _messages(marked: set[int], n: int = 6) -> list[dict[str, Any]]:
+ out = []
+ for i in range(n):
+ block = {"type": "text", "text": f"message {i}"}
+ if i in marked:
+ block["cache_control"] = CC
+ out.append({"role": "user" if i % 2 == 0 else "assistant", "content": [block]})
+ return out
+
+
+def test_under_budget_is_returned_by_identity() -> None:
+ system, messages = _system(2), _messages({4, 5})
+ s, m, t, stats = enforce_cache_breakpoint_budget(system, messages, None)
+ assert (s, m, t) == (system, messages, None)
+ assert s is system and m is messages
+ assert stats["repaired"] is False
+
+
+def test_extra_message_marker_goes_back_to_client_positions() -> None:
+ client = _messages({4, 5})
+ outbound = _messages({2, 4, 5}) # one marker a replay or transform carried in
+ s, m, _, stats = enforce_cache_breakpoint_budget(
+ _system(2), outbound, None, client_messages=client
+ )
+ assert stats["repaired"] is True
+ assert count_cache_breakpoints(s, m, None)["total"] == 4
+ marked = [i for i, msg in enumerate(m) if "cache_control" in msg["content"][0]]
+ assert marked == [4, 5]
+
+
+def test_without_client_positions_the_oldest_message_markers_go_first() -> None:
+ outbound = _messages({1, 3, 5})
+ s, m, _, stats = enforce_cache_breakpoint_budget(_system(2), outbound, None)
+ assert count_cache_breakpoints(s, m, None)["total"] == 4
+ marked = [i for i, msg in enumerate(m) if "cache_control" in msg["content"][0]]
+ # The newest marker is the write anchor for the growing tail: it stays.
+ assert marked == [3, 5]
+ assert all("cache_control" in b for b in s)
+ # Copy-on-write: the caller's objects are untouched.
+ assert "cache_control" in outbound[1]["content"][0]
+
+
+@dataclass
+class _MarkerLeakHook:
+ """Stands in for any stage that leaves an extra marker on an old message."""
+
+ name: str = "test_marker_leak"
+ savings_source: str = "test_marker_leak"
+ stream_safe: bool = True
+
+ def on_request(self, ctx: Any) -> None:
+ messages = [dict(m) for m in ctx.messages]
+ first = messages[0]
+ messages[0] = {**first, "content": [{**first["content"][0], "cache_control": CC}]}
+ ctx.messages = messages
+
+ async def on_response(self, ctx: Any, response: Any, call_model: Any) -> None:
+ return None
+
+
+def test_gateway_body_is_held_to_four_markers(headroom_client) -> None:
+ register_turn_hook(_MarkerLeakHook())
+ body = {
+ "model": "claude-sonnet-4-5",
+ "system": _system(2),
+ "messages": _messages({4, 5}),
+ "gateway": {"can_redrive": False, "can_relay_response": False},
+ }
+ resp = compress(headroom_client, body)
+ assert resp.status_code == 200, resp.text
+ data = resp.json()
+ out = data["body"]
+ assert (
+ count_cache_breakpoints(out.get("system"), out["messages"], out.get("tools"))["total"] == 4
+ )
+ assert data["messages"] == out["messages"]
+ assert "cache_breakpoint_budget" in data["transforms_applied"]
diff --git a/tests/gateway/test_contract_response_half.py b/tests/gateway/test_contract_response_half.py
index 5d5859c5d..597a0265f 100644
--- a/tests/gateway/test_contract_response_half.py
+++ b/tests/gateway/test_contract_response_half.py
@@ -29,7 +29,15 @@ from tests.gateway.conftest import compress, count_loop_tasks, registry_of
from tests.gateway.samples import big_tool_history
RESPONSE_HALF = "/v1/compress/response"
-DONE_KEYS = {"action", "turn_id", "response", "frozen_message_count", "usage_applied", "rounds"}
+DONE_KEYS = {
+ "action",
+ "turn_id",
+ "response",
+ "frozen_message_count",
+ "usage_applied",
+ "rounds",
+ "billed_usage",
+}
def _open_turn(client, session_id: str | None = "resp-s", **gateway: Any) -> dict[str, Any]:
@@ -152,6 +160,7 @@ def test_done_with_null_response_and_turn_removed(headroom_client) -> None:
assert data["response"] is None
assert data["rounds"] == 0
assert data["usage_applied"] is False # no cache signal in that usage
+ assert data["billed_usage"] == {"input_tokens": 100, "output_tokens": 7}
assert isinstance(data["frozen_message_count"], int)
assert registry_of(headroom_client).get(turn["turn_id"]) is None
again = _post(headroom_client, {"turn_id": turn["turn_id"]})
diff --git a/tests/gateway/test_redrive_loop.py b/tests/gateway/test_redrive_loop.py
index f4e5a94c4..d242dcd6a 100644
--- a/tests/gateway/test_redrive_loop.py
+++ b/tests/gateway/test_redrive_loop.py
@@ -123,11 +123,23 @@ def test_one_redrive_round_end_to_end(headroom_client, fake_provider, make_gatew
done = result.done
assert done is not None
assert done["rounds"] == 1
- # The hook's final answer IS the last provider response the gateway posted,
- # so ``response`` may be null ("forward what you already have") or that same
- # body; the client-visible invariant (asserted above) is what matters.
- assert done["response"] is None or done["response"]["id"] == "chatcmpl-final"
+ # The hook hands back the latest provider response object itself (no
+ # replacement). After a re-drive Headroom still returns that response, with
+ # usage summed over both provider calls, so the client never sees the last
+ # round's usage alone.
+ assert done["response"] is not None
+ assert done["response"]["id"] == "chatcmpl-final"
assert done["usage_applied"] is True # cached_tokens=300 on the final answer
+ # The turn's bill covers both provider calls, not just the last one.
+ assert done["billed_usage"]["input_tokens"] == 300 + 420
+ assert done["billed_usage"]["output_tokens"] == 8 + 6
+ assert done["billed_usage"]["cache_read_input_tokens"] == 300
+ assert done["response"]["usage"]["prompt_tokens"] == 720
+ assert done["response"]["usage"]["completion_tokens"] == 14
+ assert done["response"]["usage"]["prompt_tokens_details"]["cached_tokens"] == 300
+ # What the gateway hands the client is that cumulative response.
+ assert result.final_response["usage"]["prompt_tokens"] == 720
+ assert result.final_response["usage"]["completion_tokens"] == 14
assert hook.rounds_driven == 1
assert hook.queries == ["create issue"]
# L-CLEAN
@@ -221,7 +233,11 @@ def test_registry_max_redrives_env_caps_the_loop(
# ``rounds`` past the cap and is answered ``done`` instead of ``redrive``.
assert len(fake_provider.calls) == 3
assert result.done["rounds"] == 3
- assert result.done["response"] is None
+ # The capped turn still ran three provider calls for one client call: the
+ # latest provider answer comes back with usage summed over all three.
+ assert result.done["response"]["choices"][0]["message"]["tool_calls"][0]["id"] == "s2"
+ assert result.done["response"]["usage"]["prompt_tokens"] == 3 * 10
+ assert result.done["response"]["usage"]["completion_tokens"] == 3 * 5
assert registry_of(client).get(result.turn_id) is None
@@ -258,6 +274,25 @@ def test_missing_response_on_redrive_turn_is_400(headroom_client) -> None:
assert registry_of(headroom_client).get(data["turn_id"]) is not None
+def test_failed_call_closes_a_redrive_turn_without_a_response(headroom_client) -> None:
+ """L-GUARDS: a provider failure has nothing to re-drive. A gateway that
+ only knows the status (LiteLLM's failure hooks see an exception, not a
+ body) must still be able to close the turn and leave a ledger row."""
+ redrive_hook_ext.register()
+ data = headroom_client.post(
+ "/v1/compress",
+ json={**_chat_body(), "gateway": {"can_redrive": True}, "config": {"session_id": "fail"}},
+ ).json()
+ assert data["obligations"] == ["redrive"]
+ resp = headroom_client.post(
+ "/v1/compress/response", json={"turn_id": data["turn_id"], "round": 0, "status": 529}
+ )
+ assert resp.status_code == 200, resp.text
+ assert resp.json()["action"] == "done"
+ assert resp.json()["response"] is None
+ assert registry_of(headroom_client).get(data["turn_id"]) is None
+
+
def test_non_200_provider_status_skips_hooks(headroom_client, fake_provider, make_gateway) -> None:
"""L-GUARDS: a provider error is relayed, not re-driven; the turn completes."""
from tests.gateway.fake_provider import ScriptedResponse
@@ -449,3 +484,28 @@ def test_stale_retry_replays_last_redrive_without_double_billing(
assert result.path == "loop"
assert [c.request["round"] for c in result.response_half_calls] == [0, 1]
assert result.final_response["choices"][0]["message"]["content"] == "final answer"
+
+
+def test_billed_usage_is_written_in_the_responses_own_shape() -> None:
+ from headroom.proxy.gateway_turn import NormalizedUsage, _usage_in_shape
+
+ billed = NormalizedUsage(input_tokens=720, output_tokens=14, cache_read=300, cache_write=50)
+ anthropic = _usage_in_shape(
+ {"input_tokens": 420, "output_tokens": 6, "service_tier": "standard"}, billed
+ )
+ assert anthropic == {
+ "input_tokens": 720,
+ "output_tokens": 14,
+ "cache_read_input_tokens": 300,
+ "cache_creation_input_tokens": 50,
+ "service_tier": "standard",
+ }
+ chat = _usage_in_shape(
+ {"prompt_tokens": 420, "completion_tokens": 6, "total_tokens": 426}, billed
+ )
+ assert chat == {
+ "prompt_tokens": 720,
+ "completion_tokens": 14,
+ "total_tokens": 734,
+ "prompt_tokens_details": {"cached_tokens": 300},
+ }
diff --git a/tests/test_compress_session_mode.py b/tests/test_compress_session_mode.py
index cc3a23563..04a2c32d8 100644
--- a/tests/test_compress_session_mode.py
+++ b/tests/test_compress_session_mode.py
@@ -566,3 +566,77 @@ def test_usage_404_for_ttl_expired_tracker() -> None:
)
assert resp.status_code == 404
assert resp.json()["error"]["type"] == "unknown_session"
+
+
+# --------------------------------------------------------------------------- #
+# cache_control markers stay bounded across replayed turns. #
+# --------------------------------------------------------------------------- #
+
+
+def _count_message_markers(messages: list[dict]) -> int:
+ total = 0
+ for msg in messages:
+ content = msg.get("content")
+ if isinstance(content, list):
+ total += sum(1 for b in content if isinstance(b, dict) and "cache_control" in b)
+ return total
+
+
+def _anthropic_turns(n: int) -> list[dict]:
+ """An agent loop in Anthropic shape, as a caller sends it on turn n: the
+ breakpoint rides on the newest block only, moved forward every turn."""
+ messages: list[dict] = [{"role": "user", "content": [{"type": "text", "text": "fix the bug"}]}]
+ for i in range(n):
+ messages.append(
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "tool_use",
+ "id": f"t{i}",
+ "name": "bash",
+ "input": {"cmd": f"step {i}"},
+ }
+ ],
+ }
+ )
+ messages.append(
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_use_id": f"t{i}",
+ # Large and compressible, so the fresh turn rewrites it
+ # and the replay has compressed bytes to carry forward.
+ "content": _big_tool_history()[2]["content"],
+ }
+ ],
+ }
+ )
+ messages[-1]["content"][-1]["cache_control"] = {"type": "ephemeral"}
+ return messages
+
+
+def test_session_turns_do_not_accumulate_cache_control_markers() -> None:
+ # Anthropic rejects a request with more than four cache_control blocks.
+ # Across replayed turns the returned markers must track the caller's
+ # moving breakpoint, never accumulate behind it.
+ with _make_client() as client:
+ saved = 0
+ for turn in range(1, 9):
+ resp = client.post(
+ "/v1/compress",
+ json={
+ "model": "claude-sonnet-5",
+ "messages": _anthropic_turns(turn),
+ "config": {"session_id": "markers-bounded"},
+ },
+ )
+ assert resp.status_code == 200, resp.text
+ body = resp.json()
+ returned = body["messages"]
+ saved += body["tokens_saved"]
+ assert _count_message_markers(returned) == 1, (turn, _count_message_markers(returned))
+ # Precondition: compression rewrote tool results, so replay was live.
+ assert saved > 0
diff --git a/tests/test_proxy/test_kompress_background_warmup.py b/tests/test_proxy/test_kompress_background_warmup.py
new file mode 100644
index 000000000..69d68da0b
--- /dev/null
+++ b/tests/test_proxy/test_kompress_background_warmup.py
@@ -0,0 +1,76 @@
+"""Kompress loads on a background thread after startup, not inside a request.
+
+Startup cannot build the model (native init before the port binds segfaults on
+RHEL/CentOS 7-family hosts, #1908), so the first request that needed it paid the
+whole load: a gateway benchmark turn stalled 21 s. The warm-up moves that cost
+off the request path, and stays off on the affected glibc family.
+"""
+
+from __future__ import annotations
+
+import threading
+from types import SimpleNamespace
+
+import pytest
+
+from headroom.proxy.server import HeadroomProxy
+
+
+class _Kompress:
+ def __init__(self) -> None:
+ self.preloaded = threading.Event()
+
+ def preload(self, *, allow_download: bool = True) -> str:
+ self.preloaded.set()
+ return "onnx"
+
+
+def _proxy_with(compressor: object | None) -> SimpleNamespace:
+ router = SimpleNamespace(_get_kompress=lambda: compressor)
+ pipeline = SimpleNamespace(transforms=[router])
+ return SimpleNamespace(anthropic_pipeline=pipeline, openai_pipeline=pipeline)
+
+
[email protected](autouse=True)
+def _no_delay(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("HEADROOM_KOMPRESS_WARMUP_DELAY_SECONDS", "0")
+ monkeypatch.delenv("HEADROOM_KOMPRESS_WARMUP", raising=False)
+
+
+def test_warms_on_a_background_thread(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr("platform.libc_ver", lambda: ("glibc", "2.35"))
+ monkeypatch.delenv("PYTEST_CURRENT_TEST", raising=False)
+ kompress = _Kompress()
+ assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(kompress)) is True
+ assert kompress.preloaded.wait(5)
+
+
+def test_skipped_on_the_glibc_family_that_crashes(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr("platform.libc_ver", lambda: ("glibc", "2.17"))
+ monkeypatch.delenv("PYTEST_CURRENT_TEST", raising=False)
+ kompress = _Kompress()
+ assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(kompress)) is False
+ assert not kompress.preloaded.wait(0.2)
+
+
+def test_env_switch_wins_both_ways(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr("platform.libc_ver", lambda: ("glibc", "2.17"))
+ monkeypatch.setenv("HEADROOM_KOMPRESS_WARMUP", "1")
+ forced = _Kompress()
+ assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(forced)) is True
+ assert forced.preloaded.wait(5)
+
+ monkeypatch.setattr("platform.libc_ver", lambda: ("glibc", "2.35"))
+ monkeypatch.setenv("HEADROOM_KOMPRESS_WARMUP", "0")
+ off = _Kompress()
+ assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(off)) is False
+
+
+def test_no_kompress_installed_means_no_thread() -> None:
+ assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(None)) is False
+
+
+def test_test_processes_do_not_warm_unless_forced(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr("platform.libc_ver", lambda: ("glibc", "2.35"))
+ monkeypatch.setenv("PYTEST_CURRENT_TEST", "x")
+ assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(_Kompress())) is False
diff --git a/tests/test_transforms_content_router.py b/tests/test_transforms_content_router.py
index c8994cb0b..7b0211c64 100644
--- a/tests/test_transforms_content_router.py
+++ b/tests/test_transforms_content_router.py
@@ -1991,3 +1991,163 @@ def test_datetime_prefixed_user_prompt_survives_router() -> None:
result = ContentRouter().compress(prompt)
assert result.strategy_used is not CompressionStrategy.SEARCH
assert "Please update the PR desc" in result.compressed
+
+
+# --- The caller's prompt stays verbatim on replaying paths -----------------
+#
+# The proxy's coding profile turns user-message compression on so tool
+# observations inside user messages shrink. On a replaying path (proxy
+# handlers, the /v1/compress session turn) that used to reach the prompt text
+# too, and only a cache_control marker stopped it: Claude Code sets one, a
+# plain agent does not, and its task statement went out with stop words
+# stripped and pytest node ids mangled.
+
+_TASK = "Fix this bug in the library source. " * 60
+
+
+def _prompt_router(monkeypatch: pytest.MonkeyPatch) -> tuple[ContentRouter, list[str]]:
+ return _fresh_cc_router(monkeypatch)
+
+
+def test_opening_task_text_block_is_verbatim_under_replay(monkeypatch: pytest.MonkeyPatch) -> None:
+ router, _ = _prompt_router(monkeypatch)
+ messages = [{"role": "user", "content": [{"type": "text", "text": _TASK}]}]
+ out = router.apply(
+ messages,
+ _ChurnTokenizer(),
+ model_limit=100_000,
+ prefix_replay_guaranteed=True,
+ compress_user_messages=True,
+ min_tokens_to_compress=1,
+ )
+ assert out.messages[0]["content"][0]["text"] == _TASK
+
+
+def test_opening_task_string_is_verbatim_under_replay(monkeypatch: pytest.MonkeyPatch) -> None:
+ router, _ = _prompt_router(monkeypatch)
+ messages = [{"role": "user", "content": _TASK}]
+ out = router.apply(
+ messages,
+ _ChurnTokenizer(),
+ model_limit=100_000,
+ prefix_replay_guaranteed=True,
+ compress_user_messages=True,
+ min_tokens_to_compress=1,
+ )
+ assert out.messages[0]["content"] == _TASK
+
+
+def test_newest_user_turn_keeps_text_but_compresses_tool_results(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ router, calls = _prompt_router(monkeypatch)
+ follow_up = "Now also handle the async path. " * 60
+ messages = [
+ {"role": "user", "content": "run the tests"},
+ {
+ "role": "assistant",
+ "content": [{"type": "tool_use", "id": "t1", "name": "bash", "input": {}}],
+ },
+ {
+ "role": "user",
+ "content": [
+ {"type": "tool_result", "tool_use_id": "t1", "content": _FRESH_CC_OUTPUT},
+ {"type": "text", "text": follow_up},
+ ],
+ },
+ ]
+ out = router.apply(
+ messages,
+ _ChurnTokenizer(),
+ model_limit=100_000,
+ prefix_replay_guaranteed=True,
+ compress_user_messages=True,
+ min_tokens_to_compress=1,
+ )
+ blocks = out.messages[2]["content"]
+ assert blocks[0]["content"].endswith("[compressed]")
+ assert blocks[1]["text"] == follow_up
+ assert follow_up not in calls
+
+
+def test_text_harness_observation_string_still_compresses_under_replay(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ # A text harness returns its tool output as a role:user string after the
+ # assistant turn. That is an observation, not the prompt.
+ router, calls = _prompt_router(monkeypatch)
+ messages = [
+ {"role": "user", "content": "run the tests"},
+ {"role": "assistant", "content": "```bash\npytest -q\n```"},
+ {"role": "user", "content": _FRESH_CC_OUTPUT},
+ ]
+ router.apply(
+ messages,
+ _ChurnTokenizer(),
+ model_limit=100_000,
+ prefix_replay_guaranteed=True,
+ compress_user_messages=True,
+ min_tokens_to_compress=1,
+ )
+ assert _FRESH_CC_OUTPUT in calls
+
+
+def test_follow_up_prompt_string_after_assistant_turn_is_verbatim_under_replay(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ # A plain-string follow-up after an assistant reply is the caller's new
+ # instruction, the same as the list-content case above.
+ router, calls = _prompt_router(monkeypatch)
+ follow_up = "Now also handle the async path in tenacity/asyncio. " * 60
+ messages = [
+ {"role": "user", "content": "run the tests"},
+ {"role": "assistant", "content": "All 167 tests pass on the sync path."},
+ {"role": "user", "content": follow_up},
+ ]
+ out = router.apply(
+ messages,
+ _ChurnTokenizer(),
+ model_limit=100_000,
+ prefix_replay_guaranteed=True,
+ compress_user_messages=True,
+ min_tokens_to_compress=1,
+ )
+ assert out.messages[2]["content"] == follow_up
+ assert follow_up not in calls
+
+
+def test_text_harness_observation_text_block_still_compresses_under_replay(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ # The list-content twin of the string observation above: both shapes
+ # decide "prompt or observation" the same way.
+ router, calls = _prompt_router(monkeypatch)
+ messages = [
+ {"role": "user", "content": "run the tests"},
+ {"role": "assistant", "content": "```bash\npytest -q\n```"},
+ {"role": "user", "content": [{"type": "text", "text": _FRESH_CC_OUTPUT}]},
+ ]
+ router.apply(
+ messages,
+ _ChurnTokenizer(),
+ model_limit=100_000,
+ prefix_replay_guaranteed=True,
+ compress_user_messages=True,
+ min_tokens_to_compress=1,
+ )
+ assert _FRESH_CC_OUTPUT in calls
+
+
+def test_prompt_text_compresses_without_replay_guarantee(monkeypatch: pytest.MonkeyPatch) -> None:
+ # SDK / document callers that opt into user compression keep it: a
+ # spreadsheet or pasted document in a user message is the payload.
+ router, calls = _prompt_router(monkeypatch)
+ messages = [{"role": "user", "content": [{"type": "text", "text": _TASK}]}]
+ router.apply(
+ messages,
+ _ChurnTokenizer(),
+ model_limit=100_000,
+ compress_user_messages=True,
+ min_tokens_to_compress=1,
+ )
+ assert _TASK in calls
diff --git a/wiki/proxy.md b/wiki/proxy.md
index 584f4b239..69d3f558c 100644
--- a/wiki/proxy.md
+++ b/wiki/proxy.md
@@ -348,7 +348,7 @@ Optionally relay the provider's usage for attribution with `POST /v1/usage` `{"s
**Gateway mode (two-half turn contract).** Add a top-level `gateway` object (`{}` is enough) and one model turn becomes two calls, so a gateway that never lets Headroom see the provider response can still run transforms whose reload step needs it — the proxy's "no shrink without reload" rule at the API boundary. `gateway` fields: `can_redrive` (default `false`: the gateway can call the provider again with a request Headroom hands it; send `false` when streaming), `can_relay_response` (default `false`: the gateway will post status/usage after each turn), `session_affinity` (default `true`; `false` disables re-driving because pending turns are in-process), `plugin_version` (diagnostics). Every other top-level field (`system`, `tools`, `temperature`, …) passes through; `tools` is compacted deterministically (`tool_schema_compaction`; `tool_desc_compaction` with `HEADROOM_TOOL_DESC_MAX_CHARS`); extension turn hooks run, stream-safe-only unless `can_redrive` and `session_affinity` both hold. The response adds `body` (the complete provider request — forward it as-is; never contains `config`, `gateway`, `token_budget`), `turn_id`, `route` (`{model, provider, service_tier, reason}`, advisory; a routing extension's model is also written into `body.model`), `obligations` (`redrive` and/or `relay_usage`) and a `gateway` echo. Fail-open answers carry the same keys with originals and `obligations: []`. A turn is registered only when `obligations` is non-empty; with `relay_usage` the `/stats` record waits for the response half. Knobs: `HEADROOM_GATEWAY_TURN_TTL_SECONDS` (120), `HEADROOM_GATEWAY_MAX_PENDING_TURNS` (10000), `HEADROOM_GATEWAY_MAX_REDRIVES` (8). With `config.mode: "ccr"` and re-drive allowed, `headroom_retrieve` is injected into `body.tools` (`ccr_tool_injected`) and the response half answers the model's retrieval calls itself.
-`POST /v1/compress/response` (same exposure rules as `/v1/compress`) is the response half: `{"turn_id", "status": 200, "latency_ms", "usage": {...}, "response": {...}}`. `usage` accepts Anthropic, OpenAI chat (`prompt_tokens_details.cached_tokens`), OpenAI Responses (`input_tokens_details.cached_tokens`) and Kong's flat `cached_tokens` shapes, or the whole provider body (a nested `usage` key is descended into once); billed counters are summed across re-drive rounds. `response` is required when the turn carries `redrive`. Answers: `{"action": "done", "turn_id", "response": <replacement or null — forward what you hold>, "frozen_message_count", "usage_applied", "rounds"}` or `{"action": "redrive", "turn_id", "request": <full provider body to send>, "round"}` — post the provider's JSON back under the same `turn_id`; past `HEADROOM_GATEWAY_MAX_REDRIVES` the turn ends with `done` and `response: null`. Errors: 400 `invalid_request` / `missing_response`, 404 `unknown_turn` (unregistered, finished, or expired), 409 `turn_busy`.
+`POST /v1/compress/response` (same exposure rules as `/v1/compress`) is the response half: `{"turn_id", "status": 200, "latency_ms", "usage": {...}, "response": {...}}`. `usage` accepts Anthropic, OpenAI chat (`prompt_tokens_details.cached_tokens`), OpenAI Responses (`input_tokens_details.cached_tokens`) and Kong's flat `cached_tokens` shapes, or the whole provider body (a nested `usage` key is descended into once); billed counters are summed across re-drive rounds. `response` is required when the turn carries `redrive` and the provider call succeeded; a failed call closes the turn with its `status` alone. Answers: `{"action": "done", "turn_id", "response": <replacement, the latest provider response after a re-drive, or null — forward what you hold>, "frozen_message_count", "usage_applied", "rounds", "billed_usage"}` or `{"action": "redrive", "turn_id", "request": <full provider body to send>, "round"}` — post the provider's JSON back under the same `turn_id`; `billed_usage` sums every counter (cache reads and writes included) across rounds in Anthropic keys, and a re-driven turn's `response` is never null and reports that total as its `usage`; past `HEADROOM_GATEWAY_MAX_REDRIVES` the turn ends with `done` and the latest provider response. Errors: 400 `invalid_request` / `missing_response`, 404 `unknown_turn` (unregistered, finished, or expired), 409 `turn_busy`.
**Kong plugin.** [kong-plugin-headroom](https://github.com/headroomlabs-ai/kong-plugin-headroom) (its own repo; `luarocks install kong-plugin-headroom`) implements both halves for Kong Gateway 3.9 (session id from a header, usage relay from the `log` phase, re-drive loop in `access`). The contract itself is installed in Headroom through the compress-turn seam (`headroom.proxy.compress_turn`, `HEADROOM_GATEWAY_CONTRACT`), the same seam a third-party contract would use.