Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(proxy): keep Kompress warmup off the startup path (#2001)

Rod Boev10ed14e7f68e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d957e080f..54e8479e1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -121,6 +121,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
* **proxy/openai:** stop overriding an explicit client `stream_options.include_usage` on the streaming chat path. To count tokens from the trailing usage chunk, the handler set `include_usage: True` unconditionally — including flipping an explicit client `false` to `true`. The upstream then appended a usage-only chunk (`choices: []`) the client never requested, and the common `chunk.choices[0].delta` loop raised `IndexError`. The option is now only filled in when the client left the choice open (no `stream_options`, or a dict without `include_usage`); an explicit `true`/`false` is respected.
* **proxy/openai:** stop PRE_SEND from reintroducing `tools: []` after the direct #728 fix. The OpenAI request handler now mirrors the existing `tools or _original_tools is not None` body-write guard during PRE_SEND write-back, so providers that reject empty tool arrays no longer see a tools field when the client omitted it, while explicit client `tools: []` remains preserved ([#1983](https://github.com/headroomlabs-ai/headroom/issues/1983)).
* **proxy/openai:** keep the exact Responses function name `terminal` resident during OpenAI tool-search deferral so cache-mode optimization stops forwarding `terminal.terminal` and triggering the reserved-namespace 400 on Codex Responses ([#1946](https://github.com/headroomlabs-ai/headroom/issues/1946)).
+* **proxy/transforms:** startup warmup no longer calls Kompress native preload before the proxy binds its port. Enabled Kompress is deferred to first use, unavailable Kompress stays reported unavailable, and cached-model startup avoids the native ONNX load path that crashed older glibc hosts ([#1908](https://github.com/headroomlabs-ai/headroom/issues/1908)).
* **subscription/copilot:** show a fully-consumed Copilot quota as 100% used instead of unknown. `parse_copilot_quota` read `remaining = raw.get("remaining") or raw.get("quota_remaining")`, so a category reporting `remaining: 0` (quota fully spent) had that legitimate `0` treated as falsy and — with no `quota_remaining` alias in the real payload — collapsed to `None`. `CopilotQuotaCategory.used`/`used_percent` then returned `None`, so the dashboard rendered the exhausted category as `used: -` / 0% (green gauge) rather than `300/300` / 100%. Now uses an explicit `is None` check.
* **proxy/gemini:** thread the savings-profile kwargs into the native Gemini/Vertex compression paths. `handle_gemini_generate_content`, `handle_google_cloudcode_stream`, and `handle_gemini_count_tokens` called `openai_pipeline.apply()` without `proxy_pipeline_kwargs(self.config)`, so `HEADROOM_SAVINGS_PROFILE` and the ProxyConfig knobs (`target_ratio`/`min_tokens_to_compress`/`protect_recent`/`max_items_after_crush`/...) were silently dropped on the Gemini path — those requests compressed with router defaults instead of the configured profile, diverging from the Claude/Codex/Cursor paths. This is the same fix #1534 made for the OpenAI chat path; it now covers Gemini too.
* **wrap:** `headroom wrap claude` no longer installs RTK or lean-ctx by default. Claude context-tool setup is now explicit via `--context-tool`, `--no-context-tool` remains accepted, and other wrap commands keep their current defaults ([#1915](https://github.com/headroomlabs-ai/headroom/issues/1915)).
diff --git a/headroom/transforms/content_router.py b/headroom/transforms/content_router.py
index c9cddeeee..132f5cb4c 100644
--- a/headroom/transforms/content_router.py
+++ b/headroom/transforms/content_router.py
@@ -93,7 +93,7 @@ def _tool_call_args_text(raw: Any) -> str:
if isinstance(raw, str):
text = raw
elif isinstance(raw, dict):
- text = " ".join(str(v) for v in raw.values() if isinstance(v, (str, int, float, bool)))
+ text = " ".join(str(v) for v in raw.values() if isinstance(v, str | int | float | bool))
else:
return ""
return " ".join(text.split())[:300]
@@ -2704,35 +2704,17 @@ class ContentRouter(Transform):
# 1. ML text compressor: Kompress.
#
- # Eager preload is cache-only (allow_download=False): on a cold cache we
- # must NOT trigger a network download here, because this runs on the
- # blocking startup/lifespan path before the proxy binds its port. A slow
- # download stalls the bind, and a hard crash in the native download/ML
- # stack (uncatchable SIGABRT) kills the interpreter before it ever
- # listens — the proxy then "never opens its port" and the supervisor
- # gives up. When the model isn't cached we defer to first use instead.
+ # Native model initialization stays out of the blocking startup/lifespan
+ # path. The existing lazy request path loads Kompress on first use.
if self.config.enable_kompress:
- from .kompress_compressor import KompressModelNotCached
-
compressor = self._get_kompress()
if compressor:
if not hasattr(compressor, "preload"):
status["kompress"] = "enabled"
status["kompress_backend"] = "unknown"
else:
- try:
- backend = compressor.preload(allow_download=False)
- except KompressModelNotCached:
- logger.warning(
- "Kompress model not cached; compression disabled "
- "until model is downloaded. Ensure HuggingFace is "
- "accessible or pre-download with headroom-ai[ml]."
- )
- status["kompress"] = "deferred"
- else:
- logger.info("Kompress model pre-loaded at startup backend=%s", backend)
- status["kompress"] = "enabled"
- status["kompress_backend"] = str(backend)
+ logger.info("Kompress model preload deferred until first request")
+ status["kompress"] = "deferred"
else:
status["kompress"] = "unavailable"
diff --git a/tests/test_kompress_preload_deferral.py b/tests/test_kompress_preload_deferral.py
index ff32e6df4..143c61153 100644
--- a/tests/test_kompress_preload_deferral.py
+++ b/tests/test_kompress_preload_deferral.py
@@ -1,11 +1,7 @@
-"""Startup eager-preload must be cache-only so a cold cache cannot block or
-crash the proxy before it binds its port.
-
-Regression for the production crash where ``eager_load_compressors`` ran a
-network ``hf_hub_download`` of the Kompress ONNX model on the blocking
-startup/lifespan path. On a cold cache that download could hang (300s bind
-timeout) or hit a native ``SIGABRT`` in the download/ML stack, killing the
-interpreter before it ever listened on its port.
+"""Startup eager-preload must defer Kompress native loading before binding.
+
+Regression for the production crash where ``eager_load_compressors`` entered
+the cached Kompress native stack on the blocking startup/lifespan path.
"""
from __future__ import annotations
@@ -110,6 +106,12 @@ class _StubCompressor:
raise KompressModelNotCached("org/model")
+class _FatalPreloadCompressor(_StubCompressor):
+ def preload(self, *, allow_download: bool = True) -> str:
+ self.preload_calls.append(allow_download)
+ raise SystemExit("native Kompress preload")
+
+
def _router_kompress_only() -> ContentRouter:
return ContentRouter(
ContentRouterConfig(
@@ -120,24 +122,81 @@ def _router_kompress_only() -> ContentRouter:
)
-def test_eager_load_defers_when_model_not_cached(monkeypatch):
[email protected]("cache_state", ["cached", "uncached"])
+def test_eager_load_defers_kompress_regardless_of_cache_state(monkeypatch, cache_state):
router = _router_kompress_only()
- stub = _StubCompressor(cached=False)
+ stub = _StubCompressor(cached=cache_state == "cached")
monkeypatch.setattr(router, "_get_kompress", lambda: stub)
status = router.eager_load_compressors()
assert status["kompress"] == "deferred"
- assert stub.preload_calls == [False] # cache-only preload at startup
+ assert stub.preload_calls == []
+
+
+def test_eager_load_keeps_disabled_kompress_disabled(monkeypatch):
+ router = ContentRouter(
+ ContentRouterConfig(
+ enable_kompress=False,
+ enable_code_aware=False,
+ enable_smart_crusher=False,
+ )
+ )
+ stub = _StubCompressor(cached=True)
+ monkeypatch.setattr(router, "_get_kompress", lambda: stub)
+ status = router.eager_load_compressors()
+
+ assert "kompress" not in status
+ assert stub.preload_calls == []
+
+
+def test_eager_load_reports_unavailable_kompress(monkeypatch):
+ router = _router_kompress_only()
+ monkeypatch.setattr(router, "_get_kompress", lambda: None)
+
+ status = router.eager_load_compressors()
-def test_eager_load_enabled_when_model_cached(monkeypatch):
+ assert status["kompress"] == "unavailable"
+
+
+def test_non_kompress_warmups_continue_when_kompress_is_deferred(monkeypatch):
router = _router_kompress_only()
stub = _StubCompressor(cached=True)
monkeypatch.setattr(router, "_get_kompress", lambda: stub)
+ monkeypatch.setattr("headroom.compression.detector._magika_available", lambda: True)
+ monkeypatch.setattr("headroom.compression.detector._get_magika", lambda: object())
status = router.eager_load_compressors()
- assert status["kompress"] == "enabled"
- assert status["kompress_backend"] == "onnx"
- assert stub.preload_calls == [False]
+ assert status["kompress"] == "deferred"
+ assert status["magika"] == "enabled"
+ assert stub.preload_calls == []
+
+
+async def test_proxy_startup_does_not_enter_cached_kompress_native_loader(monkeypatch):
+ pytest.importorskip("httpx")
+ from headroom.proxy.server import HeadroomProxy, ProxyConfig
+
+ proxy = HeadroomProxy(
+ ProxyConfig(
+ optimize=True,
+ cache_enabled=False,
+ rate_limit_enabled=False,
+ cost_tracking_enabled=False,
+ code_aware_enabled=False,
+ )
+ )
+ router = _router_kompress_only()
+ stub = _FatalPreloadCompressor(cached=True)
+ monkeypatch.setattr(router, "_get_kompress", lambda: stub)
+ proxy.anthropic_pipeline.transforms = [router]
+ proxy.openai_pipeline.transforms = [router]
+
+ await proxy.startup()
+ try:
+ assert stub.preload_calls == []
+ assert proxy.warmup.kompress.info["source_status"] == "deferred"
+ finally:
+ await proxy.shutdown()