Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(proxy): cold-start fast pass — defer only Kompress, not the whole pipeline (#2073)
gglucassfd9ddaa238df
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 673dae2b4..d6a69a647 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -109,6 +109,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Bug Fixes+* **proxy:** run a cold-start fast pass before background-compression deferral so byte-identical freeze doesn't lock sessions to the uncompressed transcript. Since #1850, a session's provider-cached prefix is frozen in whatever form its cold start forwarded; deferring the WHOLE pipeline (`HEADROOM_BACKGROUND_COMPRESSION=1`, frozen=0, ≥50k tokens) therefore cached the raw transcript and forfeited the session's compression savings for its lifetime — including sub-second lossless wins like `read_lifecycle` stale-read drops, observed in the field as sessions permanently stuck at 0 savings. The deferral branch now runs the pipeline synchronously with the new `skip_kompress=True` kwarg (everything except the Kompress ML stage — the only stage that can blow the request budget per #1171) under a bounded fast-pass budget (`HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS`, default 10s), forwards the pruned form, and defers only Kompress to the background job (tagged `deferred:kompress_background`). Fail-open: on fast-pass timeout/error the request forwards uncompressed exactly as before. Units routed to Kompress under `skip_kompress` take the same fallback as when the model isn't ready.* **ccr:** detect `read_lifecycle` stale/superseded markers in the retrieve-tool injector so they stay redeemable. Those markers (`[Read content stale: … Retrieve original: hash=<hash>]`) store the original bytes in the CCR store under a valid hash, but none of `CCRToolInjector`'s patterns matched them — every pattern required the word "compressed" or the `<<ccr:` form. So on a frozen-prefix turn (both `read_lifecycle` and prefix freezing are on by default) the injector reported no compressed content, the `headroom_retrieve` tool was not injected, and the model was handed a marker advertising `Retrieve original: hash=X` with no tool to redeem it — silent data loss for stale reads, where retrieval is the only way to recover the original-at-read-time content (the exact case the #1006 guard exists to prevent). Added a pattern matching the load-bearing `Retrieve original: hash=` phrase, aligning the injector with the sibling `read_maturation` marker that was already (incidentally) detected.* **proxy:** strip output-only content blocks from request messages before forwarding. Anthropic's server-side refusal-fallback feature (`server-side-fallback-2026-06-01`) emits a `{"type":"fallback","from":{...},"to":{...}}` block inside the assistant response to signal that a refused request was re-served by the fallback model. That block is valid on the *response* path but rejected on the *request* path, so when a client replays the assistant turn the next request 400s (`invalid_request_error: messages.N.content.0: Input tag 'fallback' ...`) and the conversation gets permanently stuck through the proxy. `read_request_json_with_bytes` (Anthropic/OpenAI/Bedrock) and `_read_request_json` (Gemini) now drop such blocks — re-encoding the raw bytes so byte-faithful passthrough cannot leak the pre-strip body, backfilling a benign text block if a turn is emptied, and leaving requests without such blocks byte-identical (no cache churn).* **wrap/doctor:** make the Claude Remote Control gate warning accurate and stop it firing for users who never had the feature ([#1779](https://github.com/headroomlabs-ai/headroom/issues/1779)). Claude Code 2.1.196 added a client-side check that **deterministically** disables first-party Remote Control (`/remote-control` / `/rc`) whenever `ANTHROPIC_BASE_URL` points at a non-`api.anthropic.com` host — which Headroom always does. The old notice hedged ("may hide the Remote Control menu"); it now states the disable as fact, names the `/rc` command, and detects the installed Claude Code version so the wording is exact (`2.1.196` when known, `2.1.196+` when not). The gate is upstream and RC's control-plane talks to `claude.ai` (not the API host), so Headroom cannot restore it — the warning tells you to run Claude without Headroom for RC sessions. The warning is suppressed for auth modes that never had Remote Control (API-key/PAYG via `ANTHROPIC_API_KEY`/`ANTHROPIC_AUTH_TOKEN`, and Bedrock/Vertex/Foundry cloud IAM) and on Claude Code builds older than 2.1.196 where RC is unaffected by a custom base URL. Both the `headroom wrap claude` launch banner and `headroom doctor` co-report the sibling base-URL gates Headroom *does* restore — on-demand tool loading (#746, automatic) and the 1M context window (#1158, via `--1m`) — and the wrap-side co-report is session-accurate: it says "already restored via --1m" when the flag is in effect and reports tool deferral OFF (not falsely "kept on") when the user chose `--tool-search false`/`ENABLE_TOOL_SEARCH=false`; the `ENABLE_TOOL_SEARCH=...` banner line got the same accuracy fix. `is_custom_anthropic_base_url` now recognizes scheme-less values (`myproxy.local:8080`, `127.0.0.1:8787`) as custom hosts and degrades gracefully on malformed URLs instead of crashing `doctor`. `doctor` resolves the Claude Code version lazily, so runs with no custom base URL never pay the `claude --version` subprocess. No request bytes are touched (cache-safe); this is UX/notice-only.diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.pyindex 99117475a..44e851845 100644--- a/headroom/proxy/handlers/anthropic.py+++ b/headroom/proxy/handlers/anthropic.py@@ -1314,16 +1314,79 @@ class AnthropicHandlerMixin:),)- class _DeferredCompressionResult:- messages = working_messages- transforms_applied = [- "deferred:background_compression"+ # Cold-start fast pass: run everything EXCEPT the+ # Kompress ML stage synchronously before forwarding.+ # The byte-identical freeze (#1850) locks a session+ # to whatever form its cold start put in the provider+ # cache; deferring the WHOLE pipeline locks in the raw+ # transcript and forfeits the session's savings for+ # its lifetime — including sub-second wins like+ # read_lifecycle stale-read drops. Only Kompress can+ # blow the request budget (#1171), so only Kompress+ # stays deferred. Fail-open: on timeout/error the+ # request forwards exactly as before this pass+ # existed. On timeout the worker can't be cancelled+ # (Python can't preempt a running thread), but+ # _run_compression_in_executor runs it on the bounded+ # compression pool and tracks it via the leaked-thread+ # metric, so stragglers are capped and observable+ # rather than unbounded. The pass is also bounded by+ # routing + statistical crushers (observed seconds even+ # on multi-M-token counts).+ from headroom.proxy.helpers import (+ COLD_START_FAST_PASS_TIMEOUT_SECONDS,+ )++ _fast_pass = None+ try:+ async with stage_timer.measure("compression_first_stage"):+ _fast_pass = await self._run_compression_in_executor(+ lambda: self.anthropic_pipeline.apply(+ messages=working_messages,+ model=model,+ model_limit=context_limit,+ context=extract_user_query(working_messages),+ frozen_message_count=frozen_message_count,+ idle_seconds=idle_seconds,+ biases=biases,+ request_id=request_id,+ compression_policy=compression_policy,+ skip_kompress=True,+ **proxy_pipeline_kwargs(self.config),+ ),+ timeout=COLD_START_FAST_PASS_TIMEOUT_SECONDS,+ )+ except Exception as e:+ logger.info(+ "[%s] Cold-start fast pass skipped (%s: %s); "+ "deferring full pipeline to background",+ request_id,+ type(e).__name__,+ e,+ )++ if _fast_pass is not None:+ comp_cache.update_from_result(messages, _fast_pass.messages)+ _fast_pass.transforms_applied = list(+ _fast_pass.transforms_applied+ ) + [+ "deferred:kompress_background"if acceptedelse "deferred:dropped"]- timing = {}+ result = _fast_pass+ else:++ class _DeferredCompressionResult:+ messages = working_messages+ transforms_applied = [+ "deferred:background_compression"+ if accepted+ else "deferred:dropped"+ ]+ timing = {}- result = _DeferredCompressionResult()+ result = _DeferredCompressionResult()else:async with stage_timer.measure("compression_first_stage"):result = await self._run_compression_in_executor(diff --git a/headroom/proxy/helpers.py b/headroom/proxy/helpers.pyindex eb9e26e13..13fddbbcc 100644--- a/headroom/proxy/helpers.py+++ b/headroom/proxy/helpers.py@@ -620,6 +620,20 @@ try:except ValueError:COMPRESSION_TIMEOUT_SECONDS = 30.0+# Cold-start fast-pass timeout in seconds. When background compression defers+# a cold-start-large request, the handler still runs the pipeline synchronously+# with skip_kompress=True (everything except the ML stage) under this budget so+# the FORWARDED — and therefore provider-cached and byte-identically frozen —+# form carries the cheap savings instead of the raw transcript. Without the ML+# stage the pass is bounded by routing + statistical crushers (seconds, not the+# 30s Kompress budget). Fail-open: on timeout the request forwards as before.+try:+ COLD_START_FAST_PASS_TIMEOUT_SECONDS = float(+ os.environ.get("HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS", "10")+ )+except ValueError:+ COLD_START_FAST_PASS_TIMEOUT_SECONDS = 10.0+# Eager startup preload timeout in seconds. The preload (compressor/parser models,# cache-only, allow_download=False) runs off the event loop during startup; this# bound only fires on a true hang or an uncatchable native stall so the proxy stilldiff --git a/headroom/transforms/content_router.py b/headroom/transforms/content_router.pyindex 164bdb2d7..6bb34dc83 100644--- a/headroom/transforms/content_router.py+++ b/headroom/transforms/content_router.py@@ -2379,7 +2379,8 @@ class ContentRouter(Transform):# background (ensure_background_load) instead of blocking this request# thread on a 274MB download that races the compression timeout and# fails open. Until it is cached, route around the deep path.- if self.config.enable_kompress:+ # skip_kompress (cold-start fast pass) takes the identical fallback.+ if self.config.enable_kompress and not getattr(self, "_runtime_skip_kompress", False):compressor = self._get_kompress()if compressor:if not compressor.is_ready():@@ -3254,6 +3255,11 @@ class ContentRouter(Transform):self._runtime_force_kompress: bool = bool(kwargs.get("force_kompress", self.config.force_kompress_all))+ # skip_kompress: run everything EXCEPT the Kompress ML stage this+ # call. Used by the cold-start fast pass so the request-path pass+ # stays sub-second; units routed to Kompress take the same fallback+ # they take when the model isn't ready. Wins over force_kompress.+ self._runtime_skip_kompress: bool = bool(kwargs.get("skip_kompress", False))self._runtime_kompress_model: str | None = kwargs.get("kompress_model")# F2.2: capture the per-request CompressionPolicy so# ``_record_to_toin`` can gate TOIN writes ondiff --git a/tests/test_cold_start_fast_pass.py b/tests/test_cold_start_fast_pass.pynew file mode 100644index 000000000..c90e915aa--- /dev/null+++ b/tests/test_cold_start_fast_pass.py@@ -0,0 +1,287 @@+"""Cold-start fast pass: when background compression defers a cold-start-large+request, the handler still runs the pipeline synchronously with+skip_kompress=True so the FORWARDED (and therefore provider-cached,+byte-identically frozen) form carries the cheap savings. Only the Kompress ML+stage stays deferred to the background job."""++from __future__ import annotations++import json+from types import SimpleNamespace+from unittest.mock import MagicMock++import anyio+from fastapi import Request++from headroom.config import TransformResult+from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin+from headroom.proxy.models import ProxyConfig++_COMPRESSED_TEXT = "compressed tool output"+++class _DummyTokenizer:+ def count_messages(self, messages) -> int:+ return json.dumps(messages).count(" ") + 1++ def count_text(self, text: str) -> int:+ return max(1, text.count(" ") + 1)+++class _DummyMetrics:+ async def record_request(self, **kwargs):+ return None++ async def record_stage_timings(self, path, timings):+ return None++ async def record_failed(self, **kwargs):+ return None++ async def record_rate_limited(self, **kwargs):+ return None+++class _ResponseStub:+ status_code = 200+ headers: dict[str, str] = {}+ content = b'{"id":"msg_1","type":"message","role":"assistant","content":[],"usage":{"input_tokens":1,"output_tokens":1}}'++ def json(self):+ return {+ "id": "msg_1",+ "type": "message",+ "role": "assistant",+ "content": [],+ "usage": {"input_tokens": 1, "output_tokens": 1},+ }+++class _RecordingBackgroundCompressor:+ def __init__(self) -> None:+ self.enqueued: list[tuple[str, object, object]] = []++ def enqueue(self, key, compress, store) -> bool:+ self.enqueued.append((key, compress, store))+ return True+++def _fake_pipeline_apply(messages, model, **kwargs):+ compressed = []+ for msg in messages:+ new = dict(msg)+ if msg.get("role") == "user" and isinstance(msg.get("content"), list):+ new["content"] = [+ {**part, "content": _COMPRESSED_TEXT}+ if isinstance(part, dict) and part.get("type") == "tool_result"+ else part+ for part in msg["content"]+ ]+ compressed.append(new)+ return TransformResult(+ messages=compressed,+ tokens_before=1000,+ tokens_after=100,+ transforms_applied=["read_lifecycle:stale:test.py"],+ )+++class _DummyAnthropicHandler(AnthropicHandlerMixin):+ ANTHROPIC_API_URL = "https://api.anthropic.com"++ def __init__(self) -> None:+ self.rate_limiter = None+ self.metrics = _DummyMetrics()+ self.config = ProxyConfig(+ optimize=True,+ image_optimize=False,+ retry_max_attempts=1,+ retry_base_delay_ms=1,+ retry_max_delay_ms=1,+ connect_timeout_seconds=10,+ mode="token",+ cache_enabled=False,+ rate_limit_enabled=False,+ fallback_enabled=False,+ fallback_provider=None,+ prefix_freeze_enabled=False,+ memory_enabled=False,+ )+ self.usage_reporter = None+ self.anthropic_provider = SimpleNamespace(get_context_limit=lambda model: 200_000)+ self.anthropic_pipeline = SimpleNamespace(apply=MagicMock(side_effect=_fake_pipeline_apply))+ self.anthropic_backend = None+ self.cost_tracker = None+ self.memory_handler = None+ self.cache = None+ self.security = None+ self.ccr_context_tracker = None+ self.ccr_injector = None+ self.ccr_response_handler = None+ self.ccr_feedback = None+ self.ccr_batch_processor = None+ self.ccr_mcp_server = None+ self.traffic_learner = None+ self.tool_injector = None+ self.read_lifecycle_manager = None+ self.logger = SimpleNamespace(log=lambda *a, **k: None)+ self.request_logger = self.logger+ self.usage_observer = None+ self.image_compressor = None+ self.session_tracker_store = SimpleNamespace(+ compute_session_id=lambda *a, **k: "sess-1",+ get_or_create=lambda *a, **k: SimpleNamespace(+ get_frozen_message_count=lambda: 0,+ get_last_original_messages=lambda: [],+ get_last_forwarded_messages=lambda: [],+ record_request=lambda *a, **k: None,+ ),+ )+ # Cold-start deferral wiring under test.+ self._background_compression_enabled = True+ self._background_compression_min_tokens = 1+ self._background_compressor = _RecordingBackgroundCompressor()+ self.executor_calls: list[float] = []++ async def _run_compression_in_executor(self, fn, timeout):+ self.executor_calls.append(timeout)+ return fn()++ async def _next_request_id(self) -> str:+ return "req-fastpass-test"++ def _extract_tags(self, headers):+ return {}++ async def _retry_request(self, method, url, headers, body, **_kwargs):+ self.captured_body = body+ return _ResponseStub()++ def _get_compression_cache(self, session_id):+ self.comp_cache_updates: list[tuple] = getattr(self, "comp_cache_updates", [])+ return SimpleNamespace(+ apply_cached=lambda m: m,+ compute_frozen_count=lambda m: 0,+ mark_stable_from_messages=lambda *a, **k: None,+ should_defer_compression=lambda h: False,+ mark_stable=lambda h: None,+ content_hash=lambda c: "h",+ update_from_result=lambda *a: self.comp_cache_updates.append(a),+ _cache={},+ _stable_hashes=set(),+ )+++def _build_request(body: dict) -> Request:+ payload = json.dumps(body).encode("utf-8")++ async def receive():+ return {"type": "http.request", "body": payload, "more_body": False}++ scope = {+ "type": "http",+ "asgi": {"version": "3.0"},+ "http_version": "1.1",+ "method": "POST",+ "scheme": "https",+ "path": "/v1/messages",+ "raw_path": b"/v1/messages",+ "query_string": b"",+ "headers": [(b"authorization", b"Bearer sk-ant-api-test")],+ "client": ("127.0.0.1", 12345),+ "server": ("testserver", 443),+ }+ return Request(scope, receive)+++def test_cold_start_runs_fast_pass_and_defers_only_kompress(monkeypatch):+ import headroom.tokenizers as _tk++ monkeypatch.setattr(_tk, "get_tokenizer", lambda model: _DummyTokenizer())++ handler = _DummyAnthropicHandler()+ request = _build_request(+ {+ "model": "claude-3-5-sonnet-latest",+ "messages": [+ {+ "role": "user",+ "content": [+ {+ "type": "tool_result",+ "tool_use_id": "toolu_1",+ "content": "verbose stale tool output " * 200,+ }+ ],+ },+ ],+ }+ )++ anyio.run(handler.handle_anthropic_messages, request)++ # The fast pass ran synchronously with the ML stage disabled.+ assert handler.executor_calls, "fast pass never ran through the executor"+ sync_calls = [+ c for c in handler.anthropic_pipeline.apply.call_args_list if c.kwargs.get("skip_kompress")+ ]+ assert len(sync_calls) == 1, "expected exactly one synchronous skip_kompress pass"++ # The full pipeline (kompress included) went to the background queue,+ # keyed against the ORIGINAL messages for content-hash reuse.+ assert len(handler._background_compressor.enqueued) == 1+ _key, bg_compress, _store = handler._background_compressor.enqueued[0]+ bg_compress()+ bg_calls = [+ c+ for c in handler.anthropic_pipeline.apply.call_args_list+ if not c.kwargs.get("skip_kompress")+ ]+ assert len(bg_calls) == 1, "background job must run the full pipeline"++ # The FORWARDED body carries the fast-pass form — that is what the+ # provider caches and the byte-identical freeze locks in.+ forwarded = handler.captured_body["messages"]+ assert forwarded[0]["content"][0]["content"] == _COMPRESSED_TEXT++ # Fast-pass results were stored in the compression cache.+ assert handler.comp_cache_updates+++def test_fast_pass_failure_falls_back_to_full_deferral(monkeypatch):+ import headroom.tokenizers as _tk++ monkeypatch.setattr(_tk, "get_tokenizer", lambda model: _DummyTokenizer())++ handler = _DummyAnthropicHandler()++ async def _boom(fn, timeout):+ raise TimeoutError("fast pass exceeded budget")++ handler._run_compression_in_executor = _boom # type: ignore[method-assign]++ original_text = "verbose stale tool output " * 200+ request = _build_request(+ {+ "model": "claude-3-5-sonnet-latest",+ "messages": [+ {+ "role": "user",+ "content": [+ {+ "type": "tool_result",+ "tool_use_id": "toolu_1",+ "content": original_text,+ }+ ],+ },+ ],+ }+ )++ anyio.run(handler.handle_anthropic_messages, request)++ # Fail-open: original messages forwarded, background job still queued.+ forwarded = handler.captured_body["messages"]+ assert forwarded[0]["content"][0]["content"] == original_text+ assert len(handler._background_compressor.enqueued) == 1diff --git a/tests/test_transforms/test_content_router.py b/tests/test_transforms/test_content_router.pyindex a3318d24a..45ef3aea2 100644--- a/tests/test_transforms/test_content_router.py+++ b/tests/test_transforms/test_content_router.py@@ -204,6 +204,62 @@ def test_force_kompress_routes_anthropic_tool_result_to_targeted_kompress(assert captured["target_ratio"] == 0.10+def test_skip_kompress_routes_around_ml_stage(router, tokenizer, monkeypatch):+ """skip_kompress (cold-start fast pass) must never invoke the Kompress ML+ stage — units that would route there take the same fallback as when the+ model isn't ready — and wins over force_kompress."""+ calls: list[str] = []++ class FakeKompress:+ def is_ready(self) -> bool:+ return True++ def ensure_background_load(self) -> None:+ pass++ def compress(self, content, **kwargs):+ calls.append(content)+ compressed = " ".join(content.split()[:20]) + " Retrieve more: hash=deadbeef"+ return SimpleNamespace(+ compressed=compressed,+ compressed_tokens=len(compressed.split()),+ )++ monkeypatch.setattr(router, "_get_kompress", lambda: FakeKompress())+ tool_content = " ".join(+ f'{{"file":"src/module_{i}.py","line":{i},"text":"repeated search payload"}}'+ for i in range(160)+ )+ messages = [+ {+ "role": "user",+ "content": [+ {+ "type": "tool_result",+ "tool_use_id": "toolu_search_1",+ "content": tool_content,+ }+ ],+ }+ ]++ result = router.apply(+ messages,+ tokenizer,+ force_kompress=True,+ skip_kompress=True,+ target_ratio=0.10,+ compress_user_messages=True,+ min_tokens_to_compress=10,+ read_protection_window=0,+ )++ assert calls == []+ assert result.transforms_applied != ["router:tool_result:kompress"]+ # The pass still completes and returns a well-formed message list.+ assert result.messages[0]["content"][0]["content"]++def test_anthropic_tool_result_lossy_without_marker_stays_verbatim(router, tokenizer, monkeypatch):"""Reversibility gate (#1307): a lossy Kompress result on a tool_result blockwith no CCR retrieval marker is unrecoverable, so the router must keep the