Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
feat(proxy): report the same tool savings on gateway /v1/compress as the proxy path (#4467)
AlessandroSpallinaa90421083a78
diff --git a/docs/content/docs/proxy.mdx b/docs/content/docs/proxy.mdxindex 1309d2f76..cd1a7ac18 100644--- a/docs/content/docs/proxy.mdx+++ b/docs/content/docs/proxy.mdx@@ -824,6 +824,13 @@ The response keeps every field above and adds six:Fail-open answers (bypass header, empty `messages`, stateless timeout) carry the same six keys with your original body, `obligations: []` and `headers: {}`.+**Savings on a gateway answer.** The token counts follow the proxy path, so a gateway can report the same figure `/stats` does. When compaction shortened `tools`, `tokens_before` and `tokens_after` also count the tool definitions before and after it, so `tokens_saved` includes the compaction saving. Two more fields carry the tool definitions kept out of the context, which no message count sees; both are absent on fail-open answers:++| Field | Description |+| --- | --- |+| `tool_tokens_saved` | Tool-search deferral plus tools a turn hook removed, net (`tool_search_deferred_tokens` after reconciliation + `turn_hook_tools_saved_tokens` after netting against message growth). `tokens_saved + tool_tokens_saved` is the `/stats` headline for the turn. |+| `tool_tokens_deferred` | The deferral share of `tool_tokens_saved`. `/stats` books it as `estimated`: a deferred tool the model loads later is billed after all. Subtract it for a figure of measured savings only. |+When `relay_usage` is owed, the turn's `/stats` record is deferred until the response half arrives with billed input and output, cache counts, provider status and latency; a turn that never comes back is recorded as-is when it expires. Pending turns live `HEADROOM_GATEWAY_TURN_TTL_SECONDS` (default `120`, refreshed on every `redrive`), at most `HEADROOM_GATEWAY_MAX_PENDING_TURNS` at a time (default `10000`, oldest evicted first), and a turn may re-drive at most `HEADROOM_GATEWAY_MAX_REDRIVES` times (default `8`).**`config.mode: "ccr"` on this contract.** When markers were inserted and re-driving is allowed, Headroom injects the `headroom_retrieve` tool into `body.tools` (`ccr_tool_injected` in `transforms_applied`), adds the `redrive` obligation, and the response half answers the model's retrieval calls itself — the gateway just keeps re-driving. Without `can_redrive`, `ccr` mode behaves as it does today: the markers are yours to resolve.diff --git a/headroom/proxy/compress_turn.py b/headroom/proxy/compress_turn.pyindex c5f6a7826..fa4c677dc 100644--- a/headroom/proxy/compress_turn.py+++ b/headroom/proxy/compress_turn.py@@ -96,6 +96,9 @@ class CompressTurn(Protocol):# Likewise optional: ``message_growth_tokens`` (message tokens it added)# and ``hook_tool_saved_tokens`` (tool tokens it removed and booked), so# the handler can net the two before clamping the saving.+ # And ``compaction_tokens_before``/``compaction_tokens_after``, the tool+ # definitions before and after the schema compaction ``transform`` ran,+ # which the handler folds into both endpoints as the chat handlers do.def finish(self,diff --git a/headroom/proxy/gateway_extension.py b/headroom/proxy/gateway_extension.pyindex 3d61620ba..7dadc279b 100644--- a/headroom/proxy/gateway_extension.py+++ b/headroom/proxy/gateway_extension.py@@ -153,6 +153,14 @@ class GatewayTurn:def hook_tool_saved_tokens(self) -> int:return self._transformer.hook_tool_saved_tokens if self._transformer is not None else 0+ @property+ def compaction_tokens_before(self) -> int:+ return self._transformer.compaction_tokens_before if self._transformer is not None else 0++ @property+ def compaction_tokens_after(self) -> int:+ return self._transformer.compaction_tokens_after if self._transformer is not None else 0+# -- event-loop side ---------------------------------------------------def finish(self,diff --git a/headroom/proxy/gateway_turn.py b/headroom/proxy/gateway_turn.pyindex c1a91275b..bbb9206d8 100644--- a/headroom/proxy/gateway_turn.py+++ b/headroom/proxy/gateway_turn.py@@ -866,6 +866,11 @@ class RequestTransformer:# (booked on turn_hook_tools_saved_tokens), for signed netting.self.message_growth_tokens = 0self.hook_tool_saved_tokens = 0+ # Tool-definition tokens before and after schema compaction, both 0 when+ # it changed nothing or could not be counted. The compress handler folds+ # them into ``tokens_before``/``tokens_after``, as the chat handlers do.+ self.compaction_tokens_before = 0+ self.compaction_tokens_after = 0self._tokenizer: Any = Nonedef _get_tokenizer(self) -> Any:@@ -906,6 +911,15 @@ class RequestTransformer:count_tools = _count_tools_with(tokenizer)except Exception:logger.debug("gateway turn: tokenizer unavailable; hook savings uncounted")+ if tool_transforms:+ # Counted before any hook runs: a hook may edit the compacted list.+ # Tools the client already deferred are not billed, so their+ # compaction saves nothing.+ compacted_before = _safe_count(count_tools, _billed_tools(self.tools))+ compacted_after = _safe_count(count_tools, _billed_tools(tools))+ if compacted_before is not None and compacted_after is not None:+ self.compaction_tokens_before = compacted_before+ self.compaction_tokens_after = compacted_after# Built even with no hook registered: the response half drives CCR# retrieval through the same context (messages + live tools).@@ -981,6 +995,13 @@ class RequestTransformer:return messages+def _billed_tools(tools: Any) -> Any:+ """The tools that enter the billed context: all but those flagged ``defer_loading``."""+ if not isinstance(tools, list):+ return tools+ return [t for t in tools if not (isinstance(t, dict) and t.get("defer_loading"))]++def _safe_count(counter: Callable[[Any], int] | None, value: Any) -> int | None:if counter is None:return Nonediff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.pyindex 7627a0b38..5be213804 100644--- a/headroom/proxy/handlers/openai.py+++ b/headroom/proxy/handlers/openai.py@@ -54,6 +54,7 @@ from headroom.proxy.semantic_cache_key_policy import compute_request_cache_partifrom headroom.proxy.stage_timer import StageTimer, emit_stage_timings_logfrom headroom.proxy.tool_schema_savings_policy import (net_hook_tool_saving,+ tool_schema_saved_from_tags,without_deferral_flags,)from headroom.proxy.turn_hooks import run_responses_create_hooks@@ -11055,6 +11056,13 @@ class OpenAIHandlerMixin:# the saving, and offset tools the hook removed (signed, clamped once)._hook_tool_saved = 0if _turn is not None:+ # Tool-schema compaction, folded into both endpoints as the chat+ # handlers do: the message counts never see tool bytes.+ _compacted_before = int(getattr(_turn, "compaction_tokens_before", 0) or 0)+ _compacted_after = int(getattr(_turn, "compaction_tokens_after", 0) or 0)+ if 0 < _compacted_after < _compacted_before:+ tokens_before += _compacted_before+ tokens_after += _compacted_aftertokens_after += max(0, int(getattr(_turn, "tool_growth_tokens", 0) or 0))# Only a count of pre-hook messages lacks the text a hook added;# a recount of the final messages (stateless fold, session) has it.@@ -11134,6 +11142,19 @@ class OpenAIHandlerMixin:}if session_info is not None:_payload["session"] = session_info+ if _turn is not None:+ # Tool definitions kept out of the context (deferral, a hook's+ # tool shrink), which the message counts never see. The headline+ # /stats reports is tokens_saved + tool_tokens_saved. The+ # deferral share is an estimate (booked ``estimated``), so it is+ # also reported on its own for a caller that bills on measured+ # savings only.+ _payload["tool_tokens_saved"] = tool_schema_saved_from_tags(tags)+ try:+ _deferred = int(tags.get("tool_search_deferred_tokens", 0) or 0)+ except (TypeError, ValueError):+ _deferred = 0+ _payload["tool_tokens_deferred"] = max(0, _deferred)if _finished is not None and _finished.fields:_payload.update(_finished.fields)return JSONResponse(_payload)diff --git a/tests/gateway/test_hook_tool_growth.py b/tests/gateway/test_hook_tool_growth.pyindex cc91ebf50..22bc0351e 100644--- a/tests/gateway/test_hook_tool_growth.py+++ b/tests/gateway/test_hook_tool_growth.py@@ -111,14 +111,23 @@ def test_message_growth_offsets_tools_a_hook_removed(assert (net > 0) == (repeat == 5)outcome = outcomes[-1]headline = headline_tokens_saved(outcome.tokens_saved, outcome.tags)- # The gateway compacts tool schemas before hooks run, so compare with the- # hook's own measured deltas (the runner's ledger), not the raw schema.+ # The answer reports the same headline, none of it estimated.+ data = resp.json()+ assert data["tokens_saved"] + data["tool_tokens_saved"] == headline+ assert data["tool_tokens_deferred"] == 0+ # The gateway compacts tool schemas before hooks run: the hook's own ledger+ # covers the compacted schema, and the compaction delta is inside+ # tokens_saved, so the headline is the two together.+ from headroom.proxy.gateway_turn import compact_gateway_tools++ compacted, _ = compact_gateway_tools([dict(_ADDED_TOOL)])+ compaction = removed - tok.count_text(json.dumps(compacted, default=str))entries = from_tags(outcome.tags)if net > 0:(entry,) = entrieshook_net = entry["details"]["tool_tokens_saved"] + entry["details"]["message_tokens_saved"]assert entry["details"]["message_tokens_saved"] < 0- assert headline == hook_net == entry["tokens"]+ assert headline == hook_net + compaction == entry["tokens"] + compactionelse:assert entries == []assert headline == 0diff --git a/tests/gateway/test_native_deferral.py b/tests/gateway/test_native_deferral.pyindex 6f1b453ca..95d674868 100644--- a/tests/gateway/test_native_deferral.py+++ b/tests/gateway/test_native_deferral.py@@ -233,6 +233,32 @@ def test_hook_deferral_reaches_body_headers_and_outcome(assert "turn_hook_tools_saved_tokens" not in tags+def test_deferred_tokens_are_reported_beside_tokens_saved(+ headroom_client, outcome_spy, deferral_hook+) -> None:+ """The answer carries the tool tokens deferral kept out of the context, so a+ gateway can report the same headline as /stats."""+ from headroom.proxy.tool_schema_savings_policy import headline_tokens_saved++ outcomes = outcome_spy(headroom_client)+ sent = _body(tools=_many_anthropic_tools(), gateway={})+ data = compress(headroom_client, sent).json()+ n_tok = _assert_deferred(data, sent["tools"])++ assert data["tool_tokens_saved"] == n_tok+ assert data["tool_tokens_deferred"] == n_tok+ outcome = outcomes[-1]+ assert data["tokens_saved"] + data["tool_tokens_saved"] == headline_tokens_saved(+ outcome.tokens_saved, outcome.tags+ )+++def test_no_deferral_reports_no_tool_tokens_saved(headroom_client) -> None:+ data = compress(headroom_client, _body(tools=_many_anthropic_tools(), gateway={})).json()+ assert data["tool_tokens_saved"] == 0+ assert data["tool_tokens_deferred"] == 0++def test_hook_header_without_client_value_is_just_the_hooks(headroom_client, deferral_hook) -> None:data = compress(headroom_client, _body(tools=_many_anthropic_tools(), gateway={})).json()assert data["headers"] == {"anthropic-beta": BETA_TOKEN}diff --git a/tests/gateway/test_tool_compaction_accounting.py b/tests/gateway/test_tool_compaction_accounting.pynew file mode 100644index 000000000..8f50734ae--- /dev/null+++ b/tests/gateway/test_tool_compaction_accounting.py@@ -0,0 +1,85 @@+"""/v1/compress counts the tool-schema compaction it ships.++The gateway compacts tool schemas (``compact_gateway_tools``) and returns the+compacted array in ``body``, but the message counts never see tool bytes. The+chat handlers fold the compaction delta into both endpoints, so+``tokens_saved`` covers it (``tool_schema_savings_policy``: compaction "is+ALREADY inside tokens_saved"); this path must do the same.+"""++from __future__ import annotations++import json+from typing import Any++from headroom.tokenizers import get_tokenizer+from tests.gateway.conftest import compress++MODEL = "claude-sonnet-4-5"+++def _annotated_tool(i: int) -> dict[str, Any]:+ # `title` and `examples` are annotation keys schema compaction strips.+ return {+ "name": f"tool_{i}",+ "description": "Look something up.",+ "input_schema": {+ "type": "object",+ "title": f"Tool {i} input, a long title compaction removes",+ "properties": {+ "q": {+ "type": "string",+ "title": "Query",+ "examples": ["first example query", "second example query"],+ }+ },+ },+ }+++def _compress(client: Any, tools: list[dict[str, Any]]) -> dict[str, Any]:+ body = {+ "model": MODEL,+ "messages": [{"role": "user", "content": "hi"}],+ "tools": tools,+ "gateway": {"can_redrive": False, "can_relay_response": False},+ }+ resp = compress(client, body)+ assert resp.status_code == 200, resp.text+ return resp.json()+++def test_compaction_is_inside_tokens_saved(make_headroom_client) -> None:+ client = make_headroom_client(optimize=False)+ sent = [_annotated_tool(i) for i in range(3)]+ data = _compress(client, sent)++ assert "tool_schema_compaction" in data["transforms_applied"]+ tok = get_tokenizer(MODEL)+ shipped = data["body"]["tools"]+ removed = tok.count_text(json.dumps(sent, default=str)) - tok.count_text(+ json.dumps(shipped, default=str)+ )+ assert removed > 0+ # optimize=False leaves the messages alone: the whole saving is the tools.+ assert data["tokens_saved"] == removed+ assert data["tokens_before"] - data["tokens_after"] == data["tokens_saved"]+++def test_tools_compaction_leaves_alone_count_nothing(make_headroom_client) -> None:+ client = make_headroom_client(optimize=False)+ plain = {"name": "plain", "description": "x", "input_schema": {"type": "object"}}+ data = _compress(client, [plain])++ assert "tool_schema_compaction" not in data["transforms_applied"]+ assert data["tokens_saved"] == 0+++def test_compacting_client_deferred_tools_counts_nothing(make_headroom_client) -> None:+ # A tool the client marked defer_loading is not in the billed context.+ client = make_headroom_client(optimize=False)+ deferred = [{**_annotated_tool(i), "defer_loading": True} for i in range(3)]+ data = _compress(client, deferred)++ assert "tool_schema_compaction" in data["transforms_applied"]+ assert data["tokens_saved"] == 0diff --git a/wiki/proxy.md b/wiki/proxy.mdindex 5d56ccff1..6862ee842 100644--- a/wiki/proxy.md+++ b/wiki/proxy.md@@ -346,7 +346,7 @@ Note `protect_recent` is not a substitute — it guards the newest messages, whiOptionally relay the provider's usage for attribution with `POST /v1/usage` `{"session_id": "...", "usage": {...}}`. Anthropic `cache_read_input_tokens` / `cache_creation_input_tokens` and OpenAI `prompt_tokens_details.cached_tokens` / flat `cached_tokens` are accepted; a block with none of them is a 400. Replies `{"session_id", "frozen_message_count", "applied"}` (`applied: false, reason: "no_cache_signal"` when the only present field is 0), 404 `unknown_session`, or 503 `session_busy` (retry). Telemetry only — it never raises the frozen count.-**Gateway mode (two-half turn contract).** Add a top-level `gateway` object (`{}` is enough) and one model turn becomes two calls, so a gateway that never lets Headroom see the provider response can still run transforms whose reload step needs it — the proxy's "no shrink without reload" rule at the API boundary. `gateway` fields: `can_redrive` (default `false`: the gateway can call the provider again with a request Headroom hands it; send `false` when streaming), `can_relay_response` (default `false`: the gateway will post status/usage after each turn), `session_affinity` (default `true`; `false` disables re-driving because pending turns are in-process), `plugin_version` (diagnostics). Every other top-level field (`system`, `tools`, `temperature`, …) passes through; `tools` is compacted deterministically (`tool_schema_compaction`; `tool_desc_compaction` with `HEADROOM_TOOL_DESC_MAX_CHARS`); extension turn hooks run, stream-safe-only unless `can_redrive` and `session_affinity` both hold. The response adds `body` (the complete provider request — forward it as-is; never contains `config`, `gateway`, `token_budget`), `turn_id`, `route` (`{model, provider, service_tier, reason}`, advisory; a routing extension's model is also written into `body.model`), `obligations` (`redrive` and/or `relay_usage`) and a `gateway` echo. Fail-open answers carry the same keys with originals and `obligations: []`. A turn is registered only when `obligations` is non-empty; with `relay_usage` the `/stats` record waits for the response half. Knobs: `HEADROOM_GATEWAY_TURN_TTL_SECONDS` (120), `HEADROOM_GATEWAY_MAX_PENDING_TURNS` (10000), `HEADROOM_GATEWAY_MAX_REDRIVES` (8). With `config.mode: "ccr"` and re-drive allowed, `headroom_retrieve` is injected into `body.tools` (`ccr_tool_injected`) and the response half answers the model's retrieval calls itself.+**Gateway mode (two-half turn contract).** Add a top-level `gateway` object (`{}` is enough) and one model turn becomes two calls, so a gateway that never lets Headroom see the provider response can still run transforms whose reload step needs it — the proxy's "no shrink without reload" rule at the API boundary. `gateway` fields: `can_redrive` (default `false`: the gateway can call the provider again with a request Headroom hands it; send `false` when streaming), `can_relay_response` (default `false`: the gateway will post status/usage after each turn), `session_affinity` (default `true`; `false` disables re-driving because pending turns are in-process), `plugin_version` (diagnostics). Every other top-level field (`system`, `tools`, `temperature`, …) passes through; `tools` is compacted deterministically (`tool_schema_compaction`; `tool_desc_compaction` with `HEADROOM_TOOL_DESC_MAX_CHARS`); extension turn hooks run, stream-safe-only unless `can_redrive` and `session_affinity` both hold. The response adds `body` (the complete provider request — forward it as-is; never contains `config`, `gateway`, `token_budget`), `turn_id`, `route` (`{model, provider, service_tier, reason}`, advisory; a routing extension's model is also written into `body.model`), `obligations` (`redrive` and/or `relay_usage`) and a `gateway` echo. Fail-open answers carry the same keys with originals and `obligations: []`. Savings follow the proxy path: when compaction shortened `tools`, `tokens_before`/`tokens_after` also count the tool definitions, and the answer adds `tool_tokens_saved` (tool-search deferral plus tools a turn hook removed, both net; `tokens_saved + tool_tokens_saved` is the `/stats` headline) and `tool_tokens_deferred` (its deferral share, booked `estimated`); both are absent on fail-open answers. A turn is registered only when `obligations` is non-empty; with `relay_usage` the `/stats` record waits for the response half. Knobs: `HEADROOM_GATEWAY_TURN_TTL_SECONDS` (120), `HEADROOM_GATEWAY_MAX_PENDING_TURNS` (10000), `HEADROOM_GATEWAY_MAX_REDRIVES` (8). With `config.mode: "ccr"` and re-drive allowed, `headroom_retrieve` is injected into `body.tools` (`ccr_tool_injected`) and the response half answers the model's retrieval calls itself.`POST /v1/compress/response` (same exposure rules as `/v1/compress`) is the response half: `{"turn_id", "status": 200, "latency_ms", "usage": {...}, "response": {...}}`. `usage` accepts Anthropic, OpenAI chat (`prompt_tokens_details.cached_tokens`), OpenAI Responses (`input_tokens_details.cached_tokens`) and Kong's flat `cached_tokens` shapes, or the whole provider body (a nested `usage` key is descended into once); billed counters are summed across re-drive rounds. `response` is required when the turn carries `redrive` and the provider call succeeded; a failed call closes the turn with its `status` alone. Answers: `{"action": "done", "turn_id", "response": <replacement, the latest provider response after a re-drive, or null — forward what you hold>, "frozen_message_count", "usage_applied", "rounds", "billed_usage"}` or `{"action": "redrive", "turn_id", "request": <full provider body to send>, "round"}` — post the provider's JSON back under the same `turn_id`; `billed_usage` sums every counter (cache reads and writes included) across rounds in Anthropic keys, and a re-driven turn's `response` is never null and reports that total as its `usage`; past `HEADROOM_GATEWAY_MAX_REDRIVES` the turn ends with `done` and the latest provider response. Errors: 400 `invalid_request` / `missing_response`, 404 `unknown_turn` (unregistered, finished, or expired), 409 `turn_busy`.