Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
perf(proxy): offload image compression off event loop (#1612)
inix7fe203cfa103
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 82d25a32f..c2bdb938c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -89,6 +89,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0* **install:** stop duplicating the container ENTRYPOINT in the `persistent-docker` runtime command. The published image already runs `headroom proxy` as its ENTRYPOINT, but `build_runtime_command` re-added `headroom proxy` after the image name, so the container ran `headroom proxy headroom proxy --host 0.0.0.0 …` and Click aborted with "Got unexpected extra arguments (headroom proxy)" — the deployment never became ready and rollback left nothing running. The runtime command now appends only the proxy flags ([#833](https://github.com/chopratejas/headroom/issues/833)).* **proxy:** retry upstream `529 overloaded_error` like a 429 on both the streaming and non-streaming forwarders, honoring `Retry-After`. The streaming path previously surfaced a 529 straight to the client with no retry (interactive sessions saw "Overloaded" immediately), and `_retry_request` retried it only via the generic 5xx path — raising on exhaustion instead of returning the 529 verbatim, and ignoring `Retry-After`. A shared `RETRYABLE_OVERLOAD_STATUSES = {429, 529}` keeps the two forwarders in agreement (extends [#1221](https://github.com/headroomlabs-ai/headroom/issues/1221)).* **gemini:** run compression off the asyncio event loop. The Gemini handlers (`generateContent`, Cloud Code stream, `countTokens`) ran the CPU-bound compression pipeline (Magika detection plus ML compression) synchronously on the loop, stalling every concurrent request for the duration of each Gemini request's compression. They now offload it via the shared compression executor, matching the existing OpenAI and Anthropic paths.+* **proxy:** run image compression off the asyncio event loop. The Anthropic and OpenAI handlers ran the CPU-bound image compressor (ONNX technique routing plus Pillow resize and OCR) synchronously on the loop, stalling every concurrent request for the duration of each image request's compression. They now offload it via the shared compression executor with a timeout and fail open on error, matching the existing text-compression path.* **proxy:** queue mid-turn user messages on non-Bedrock streaming path instead of silently dropping them — closes [#902](https://github.com/headroomlabs-ai/headroom/issues/902).* **proxy:** add `--protect-tool-results` / `HEADROOM_PROTECT_TOOL_RESULTS` to prevent lossy compression of exact-output tool results (e.g. `Bash cat`/`grep` results) — closes [#1307](https://github.com/headroomlabs-ai/headroom/issues/1307).* **cli:** add `--rpm`/`--tpm` and `HEADROOM_RPM`/`HEADROOM_TPM` to the Click proxy command for rate-limit parity with the legacy CLI -- closes [#1350](https://github.com/headroomlabs-ai/headroom/issues/1350) (Problem 1).diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.pyindex 7d6ddea7b..116b6a01e 100644--- a/headroom/proxy/handlers/anthropic.py+++ b/headroom/proxy/handlers/anthropic.py@@ -1030,11 +1030,18 @@ class AnthropicHandlerMixin:)_image_decision.apply_to_tags(tags)if _image_decision.should_compress and not is_cache_mode(self.config.mode):+ from headroom.proxy.helpers import COMPRESSION_TIMEOUT_SECONDS+compressor = Nonetry:compressor = _get_image_compressor()if compressor and compressor.has_images(messages):- messages = compressor.compress(messages, provider="anthropic")+ # Offload CPU-bound image compression onto the bounded+ # executor (same as text compression); inline blocked the loop.+ messages = await self._run_compression_in_executor(+ lambda: compressor.compress(messages, provider="anthropic"),+ timeout=COMPRESSION_TIMEOUT_SECONDS,+ )body_mutation_tracker.mark_mutated("image_compression")if compressor.last_result:logger.info(@@ -1043,6 +1050,10 @@ class AnthropicHandlerMixin:f"{compressor.last_result.original_tokens} -> "f"{compressor.last_result.compressed_tokens} tokens)")+ except Exception as e:+ # Image compression is best-effort — fail open on timeout/error and+ # forward the original messages, matching the text path.+ logger.warning(f"Image compression failed: {type(e).__name__}: {e}")finally:if compressor and hasattr(compressor, "close"):compressor.close()diff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.pyindex 11de719ac..1460b9db2 100644--- a/headroom/proxy/handlers/openai.py+++ b/headroom/proxy/handlers/openai.py@@ -1743,7 +1743,12 @@ class OpenAIHandlerMixin:try:compressor = _get_image_compressor()if compressor and compressor.has_images(messages):- messages = compressor.compress(messages, provider="openai")+ # Offload CPU-bound image compression onto the bounded+ # executor (same as text compression); inline blocked the loop.+ messages = await self._run_compression_in_executor(+ lambda: compressor.compress(messages, provider="openai"),+ timeout=COMPRESSION_TIMEOUT_SECONDS,+ )if compressor.last_result:logger.info(f"[{request_id}] Image: {compressor.last_result.technique.value} "@@ -1751,6 +1756,10 @@ class OpenAIHandlerMixin:f"{compressor.last_result.original_tokens} → "f"{compressor.last_result.compressed_tokens} tokens)")+ except Exception as e:+ # Image compression is best-effort — fail open on timeout/error and+ # forward the original messages, matching the text path.+ logger.warning(f"[{request_id}] Image compression failed: {type(e).__name__}: {e}")finally:if compressor and hasattr(compressor, "close"):compressor.close()diff --git a/tests/test_image_compression_offload.py b/tests/test_image_compression_offload.pynew file mode 100644index 000000000..0794bb05e--- /dev/null+++ b/tests/test_image_compression_offload.py@@ -0,0 +1,94 @@+"""Image compression offload (perf): the Anthropic and OpenAI handlers must run the+CPU-bound `ImageCompressor.compress()` on the bounded compression executor, not inline+on the event loop, and fail open if it raises — matching the text-compression path.++Mirrors test_gemini_compression_offload.py. The wiring (each image block awaits+`_run_compression_in_executor(lambda: compressor.compress(...))`) reuses the proven+text path; these tests assert the observable properties that wiring delivers: the+blocks are async + offloaded + fail open, and the executor keeps the loop responsive+while a slow compression runs on a worker thread.+"""++from __future__ import annotations++import asyncio+import inspect+import threading+import time++from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin+from headroom.proxy.handlers.openai import OpenAIHandlerMixin+from headroom.proxy.server import ProxyConfig, create_app+++def _make_proxy(): # noqa: ANN202 — returns the internal HeadroomProxy+ app = create_app(+ ProxyConfig(+ optimize=True,+ cache_enabled=False,+ rate_limit_enabled=False,+ cost_tracking_enabled=False,+ )+ )+ return app.state.proxy+++def test_image_blocks_offload_compress_and_fail_open() -> None:+ """Each image-compress block must be async, offload compress() onto the executor with a+ timeout, and fail open. Guards a future refactor from silently re-inlining compress()."""+ for mixin, method in (+ (AnthropicHandlerMixin, "handle_anthropic_messages"),+ (OpenAIHandlerMixin, "handle_openai_chat"),+ ):+ fn = getattr(mixin, method)+ assert inspect.iscoroutinefunction(fn), f"{method} must be async to await the offload"+ src = inspect.getsource(fn)+ assert "compressor.compress(" in src, f"{method}: image compress call missing"+ assert "_run_compression_in_executor(" in src, f"{method}: compress not offloaded"+ assert "COMPRESSION_TIMEOUT_SECONDS" in src, f"{method}: offload missing a timeout"+ assert "Image compression failed" in src, f"{method}: image compress not fail-open"+++async def test_image_compress_offload_runs_on_worker_thread() -> None:+ """The exact call the image blocks make — _run_compression_in_executor(lambda: compress()) —+ runs compress() on a 'headroom-compress' executor thread, not the event-loop thread."""+ proxy = _make_proxy()+ loop_thread = threading.current_thread().name+ seen: dict[str, str] = {}++ def _slow_compress(): # noqa: ANN202 — stands in for ImageCompressor.compress+ seen["thread"] = threading.current_thread().name+ time.sleep(0.1)+ return [{"role": "user", "content": "compressed"}]++ result = await proxy._run_compression_in_executor(_slow_compress, timeout=10)++ assert result == [{"role": "user", "content": "compressed"}]+ assert seen["thread"].startswith("headroom-compress")+ assert seen["thread"] != loop_thread+++async def test_image_compress_offload_keeps_event_loop_responsive() -> None:+ """While a slow image compression runs on the executor, the loop keeps scheduling+ coroutines. The bug this fixes — a bare inline compress() — would starve them to ~0 ticks."""+ proxy = _make_proxy()+ ticks = 0++ async def _ticker() -> None:+ nonlocal ticks+ while True:+ await asyncio.sleep(0.01)+ ticks += 1++ def _slow_compress(): # noqa: ANN202+ time.sleep(0.3)+ return "x"++ tick_task = asyncio.create_task(_ticker())+ try:+ result = await proxy._run_compression_in_executor(_slow_compress, timeout=10)+ finally:+ tick_task.cancel()++ assert result == "x"+ assert ticks >= 5 # ~30 expected at 10ms over 0.3s; a blocked loop yields near zero