Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(proxy): keep Kompress warmup off the startup path (#2001)
Rod Boev10ed14e7f68e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d957e080f..54e8479e1 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -121,6 +121,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0* **proxy/openai:** stop overriding an explicit client `stream_options.include_usage` on the streaming chat path. To count tokens from the trailing usage chunk, the handler set `include_usage: True` unconditionally — including flipping an explicit client `false` to `true`. The upstream then appended a usage-only chunk (`choices: []`) the client never requested, and the common `chunk.choices[0].delta` loop raised `IndexError`. The option is now only filled in when the client left the choice open (no `stream_options`, or a dict without `include_usage`); an explicit `true`/`false` is respected.* **proxy/openai:** stop PRE_SEND from reintroducing `tools: []` after the direct #728 fix. The OpenAI request handler now mirrors the existing `tools or _original_tools is not None` body-write guard during PRE_SEND write-back, so providers that reject empty tool arrays no longer see a tools field when the client omitted it, while explicit client `tools: []` remains preserved ([#1983](https://github.com/headroomlabs-ai/headroom/issues/1983)).* **proxy/openai:** keep the exact Responses function name `terminal` resident during OpenAI tool-search deferral so cache-mode optimization stops forwarding `terminal.terminal` and triggering the reserved-namespace 400 on Codex Responses ([#1946](https://github.com/headroomlabs-ai/headroom/issues/1946)).+* **proxy/transforms:** startup warmup no longer calls Kompress native preload before the proxy binds its port. Enabled Kompress is deferred to first use, unavailable Kompress stays reported unavailable, and cached-model startup avoids the native ONNX load path that crashed older glibc hosts ([#1908](https://github.com/headroomlabs-ai/headroom/issues/1908)).* **subscription/copilot:** show a fully-consumed Copilot quota as 100% used instead of unknown. `parse_copilot_quota` read `remaining = raw.get("remaining") or raw.get("quota_remaining")`, so a category reporting `remaining: 0` (quota fully spent) had that legitimate `0` treated as falsy and — with no `quota_remaining` alias in the real payload — collapsed to `None`. `CopilotQuotaCategory.used`/`used_percent` then returned `None`, so the dashboard rendered the exhausted category as `used: -` / 0% (green gauge) rather than `300/300` / 100%. Now uses an explicit `is None` check.* **proxy/gemini:** thread the savings-profile kwargs into the native Gemini/Vertex compression paths. `handle_gemini_generate_content`, `handle_google_cloudcode_stream`, and `handle_gemini_count_tokens` called `openai_pipeline.apply()` without `proxy_pipeline_kwargs(self.config)`, so `HEADROOM_SAVINGS_PROFILE` and the ProxyConfig knobs (`target_ratio`/`min_tokens_to_compress`/`protect_recent`/`max_items_after_crush`/...) were silently dropped on the Gemini path — those requests compressed with router defaults instead of the configured profile, diverging from the Claude/Codex/Cursor paths. This is the same fix #1534 made for the OpenAI chat path; it now covers Gemini too.* **wrap:** `headroom wrap claude` no longer installs RTK or lean-ctx by default. Claude context-tool setup is now explicit via `--context-tool`, `--no-context-tool` remains accepted, and other wrap commands keep their current defaults ([#1915](https://github.com/headroomlabs-ai/headroom/issues/1915)).diff --git a/headroom/transforms/content_router.py b/headroom/transforms/content_router.pyindex c9cddeeee..132f5cb4c 100644--- a/headroom/transforms/content_router.py+++ b/headroom/transforms/content_router.py@@ -93,7 +93,7 @@ def _tool_call_args_text(raw: Any) -> str:if isinstance(raw, str):text = rawelif isinstance(raw, dict):- text = " ".join(str(v) for v in raw.values() if isinstance(v, (str, int, float, bool)))+ text = " ".join(str(v) for v in raw.values() if isinstance(v, str | int | float | bool))else:return ""return " ".join(text.split())[:300]@@ -2704,35 +2704,17 @@ class ContentRouter(Transform):# 1. ML text compressor: Kompress.#- # Eager preload is cache-only (allow_download=False): on a cold cache we- # must NOT trigger a network download here, because this runs on the- # blocking startup/lifespan path before the proxy binds its port. A slow- # download stalls the bind, and a hard crash in the native download/ML- # stack (uncatchable SIGABRT) kills the interpreter before it ever- # listens — the proxy then "never opens its port" and the supervisor- # gives up. When the model isn't cached we defer to first use instead.+ # Native model initialization stays out of the blocking startup/lifespan+ # path. The existing lazy request path loads Kompress on first use.if self.config.enable_kompress:- from .kompress_compressor import KompressModelNotCached-compressor = self._get_kompress()if compressor:if not hasattr(compressor, "preload"):status["kompress"] = "enabled"status["kompress_backend"] = "unknown"else:- try:- backend = compressor.preload(allow_download=False)- except KompressModelNotCached:- logger.warning(- "Kompress model not cached; compression disabled "- "until model is downloaded. Ensure HuggingFace is "- "accessible or pre-download with headroom-ai[ml]."- )- status["kompress"] = "deferred"- else:- logger.info("Kompress model pre-loaded at startup backend=%s", backend)- status["kompress"] = "enabled"- status["kompress_backend"] = str(backend)+ logger.info("Kompress model preload deferred until first request")+ status["kompress"] = "deferred"else:status["kompress"] = "unavailable"diff --git a/tests/test_kompress_preload_deferral.py b/tests/test_kompress_preload_deferral.pyindex ff32e6df4..143c61153 100644--- a/tests/test_kompress_preload_deferral.py+++ b/tests/test_kompress_preload_deferral.py@@ -1,11 +1,7 @@-"""Startup eager-preload must be cache-only so a cold cache cannot block or-crash the proxy before it binds its port.--Regression for the production crash where ``eager_load_compressors`` ran a-network ``hf_hub_download`` of the Kompress ONNX model on the blocking-startup/lifespan path. On a cold cache that download could hang (300s bind-timeout) or hit a native ``SIGABRT`` in the download/ML stack, killing the-interpreter before it ever listened on its port.+"""Startup eager-preload must defer Kompress native loading before binding.++Regression for the production crash where ``eager_load_compressors`` entered+the cached Kompress native stack on the blocking startup/lifespan path."""from __future__ import annotations@@ -110,6 +106,12 @@ class _StubCompressor:raise KompressModelNotCached("org/model")+class _FatalPreloadCompressor(_StubCompressor):+ def preload(self, *, allow_download: bool = True) -> str:+ self.preload_calls.append(allow_download)+ raise SystemExit("native Kompress preload")++def _router_kompress_only() -> ContentRouter:return ContentRouter(ContentRouterConfig(@@ -120,24 +122,81 @@ def _router_kompress_only() -> ContentRouter:)-def test_eager_load_defers_when_model_not_cached(monkeypatch):[email protected]("cache_state", ["cached", "uncached"])+def test_eager_load_defers_kompress_regardless_of_cache_state(monkeypatch, cache_state):router = _router_kompress_only()- stub = _StubCompressor(cached=False)+ stub = _StubCompressor(cached=cache_state == "cached")monkeypatch.setattr(router, "_get_kompress", lambda: stub)status = router.eager_load_compressors()assert status["kompress"] == "deferred"- assert stub.preload_calls == [False] # cache-only preload at startup+ assert stub.preload_calls == []+++def test_eager_load_keeps_disabled_kompress_disabled(monkeypatch):+ router = ContentRouter(+ ContentRouterConfig(+ enable_kompress=False,+ enable_code_aware=False,+ enable_smart_crusher=False,+ )+ )+ stub = _StubCompressor(cached=True)+ monkeypatch.setattr(router, "_get_kompress", lambda: stub)+ status = router.eager_load_compressors()++ assert "kompress" not in status+ assert stub.preload_calls == []+++def test_eager_load_reports_unavailable_kompress(monkeypatch):+ router = _router_kompress_only()+ monkeypatch.setattr(router, "_get_kompress", lambda: None)++ status = router.eager_load_compressors()-def test_eager_load_enabled_when_model_cached(monkeypatch):+ assert status["kompress"] == "unavailable"+++def test_non_kompress_warmups_continue_when_kompress_is_deferred(monkeypatch):router = _router_kompress_only()stub = _StubCompressor(cached=True)monkeypatch.setattr(router, "_get_kompress", lambda: stub)+ monkeypatch.setattr("headroom.compression.detector._magika_available", lambda: True)+ monkeypatch.setattr("headroom.compression.detector._get_magika", lambda: object())status = router.eager_load_compressors()- assert status["kompress"] == "enabled"- assert status["kompress_backend"] == "onnx"- assert stub.preload_calls == [False]+ assert status["kompress"] == "deferred"+ assert status["magika"] == "enabled"+ assert stub.preload_calls == []+++async def test_proxy_startup_does_not_enter_cached_kompress_native_loader(monkeypatch):+ pytest.importorskip("httpx")+ from headroom.proxy.server import HeadroomProxy, ProxyConfig++ proxy = HeadroomProxy(+ ProxyConfig(+ optimize=True,+ cache_enabled=False,+ rate_limit_enabled=False,+ cost_tracking_enabled=False,+ code_aware_enabled=False,+ )+ )+ router = _router_kompress_only()+ stub = _FatalPreloadCompressor(cached=True)+ monkeypatch.setattr(router, "_get_kompress", lambda: stub)+ proxy.anthropic_pipeline.transforms = [router]+ proxy.openai_pipeline.transforms = [router]++ await proxy.startup()+ try:+ assert stub.preload_calls == []+ assert proxy.warmup.kompress.info["source_status"] == "deferred"+ finally:+ await proxy.shutdown()