Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(proxy/batch): preserve sibling tool configs on Google batch requests (#2177)

Abhay Singha5d7e12c9078
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 88fd16344..8147af13c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -116,6 +116,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Bug Fixes
* **proxy:** run a cold-start fast pass before background-compression deferral so byte-identical freeze doesn't lock sessions to the uncompressed transcript. Since #1850, a session's provider-cached prefix is frozen in whatever form its cold start forwarded; deferring the WHOLE pipeline (`HEADROOM_BACKGROUND_COMPRESSION=1`, frozen=0, ≥50k tokens) therefore cached the raw transcript and forfeited the session's compression savings for its lifetime — including sub-second lossless wins like `read_lifecycle` stale-read drops, observed in the field as sessions permanently stuck at 0 savings. The deferral branch now runs the pipeline synchronously with the new `skip_kompress=True` kwarg (everything except the Kompress ML stage — the only stage that can blow the request budget per #1171) under a bounded fast-pass budget (`HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS`, default 10s), forwards the pruned form, and defers only Kompress to the background job (tagged `deferred:kompress_background`). Fail-open: on fast-pass timeout/error the request forwards uncompressed exactly as before. Units routed to Kompress under `skip_kompress` take the same fallback as when the model isn't ready.
+* **proxy/batch:** preserve sibling tool configs on Google batch requests. When optimizing a Gemini batch item, the handler rebuilt the request's `tools` as a single `[{"functionDeclarations": ...}]` entry, discarding any other entries in the array (`googleSearch`, `codeExecution`, etc.). A batch request that combined function calling with Google Search or code execution therefore reached Google with those siblings stripped, silently disabling them. The handler now replaces only the `functionDeclarations` entry (appending one when the original had none) and keeps every sibling entry intact.
* **savings:** record the pre-compression original as the ledger `before`, not the forwarded count. In `PrometheusMetrics.record_request` the durable savings ledger was written with `tokens_before=input_tokens`, but `input_tokens` there is the optimized (post-compression) count that was forwarded (`emit_request_outcome` passes `outcome.optimized_tokens`). `headroom savings` derives the reported reduction percent as saved / before, so understating `before` by `tokens_saved` inflated it — a real 40% reduction (1000 → 600) was reported as ~67% (400 / 600). The event now records `tokens_before=input_tokens + tokens_saved` (the reconstructed original) and `tokens_after=input_tokens` (the forwarded count); the `saved` and cost figures are unchanged.
* **proxy/gemini:** forward a non-JSON upstream error body with its real status instead of a synthetic 502. In `handle_gemini_generate_content` the token-extraction `except (KeyError, TypeError, AttributeError)` guarding `response.json()` omitted `json.JSONDecodeError` / `ValueError`, so a non-JSON body (an HTML/empty error page from an overloaded Google/Vertex/Copilot frontend, common on 5xx/429) escaped to the outer `except Exception` and was returned as a generic 502 — discarding the real upstream `status_code` and body and defeating the client's retry/backoff. The except now catches the JSON/ValueError family, matching the all-non-text sibling branch, so the true status and body are forwarded verbatim.
* **init/codex:** don't overwrite the user's `hooks.json`. `_ensure_codex_hooks` wrote a fresh payload containing only Headroom's two hooks, wholesale-replacing `~/.codex/hooks.json` — so any user-managed Codex hooks (and other top-level keys) were silently destroyed on `headroom init codex`. It now read-merges: existing entries are preserved, Headroom's are deduped on the `headroom-init-codex` marker and appended, matching `_ensure_claude_hooks` / `_ensure_copilot_hooks`.
diff --git a/headroom/proxy/handlers/batch.py b/headroom/proxy/handlers/batch.py
index 928467be2..3681294a4 100644
--- a/headroom/proxy/handlers/batch.py
+++ b/headroom/proxy/handlers/batch.py
@@ -242,7 +242,24 @@ class BatchHandlerMixin:
if optimized_sys_inst:
compressed_req_content["systemInstruction"] = optimized_sys_inst
if existing_funcs is not None:
- compressed_req_content["tools"] = [{"functionDeclarations": existing_funcs}]
+ # Preserve sibling tool configs (googleSearch, codeExecution,
+ # ...) that live alongside functionDeclarations in the tools
+ # array. Replace only the functionDeclarations entry with the
+ # (possibly CCR-injected) funcs and append a new entry when the
+ # original had none, instead of collapsing the whole array to a
+ # single functionDeclarations entry (which dropped the siblings
+ # and silently disabled Google Search / code execution).
+ rebuilt_tools = []
+ replaced = False
+ for tool in tools or []:
+ if "functionDeclarations" in tool:
+ rebuilt_tools.append({**tool, "functionDeclarations": existing_funcs})
+ replaced = True
+ else:
+ rebuilt_tools.append(tool)
+ if not replaced:
+ rebuilt_tools.append({"functionDeclarations": existing_funcs})
+ compressed_req_content["tools"] = rebuilt_tools
compressed_req = {
"request": compressed_req_content,
diff --git a/tests/test_proxy_handlers_batch.py b/tests/test_proxy_handlers_batch.py
index 30011017e..79c07182e 100644
--- a/tests/test_proxy_handlers_batch.py
+++ b/tests/test_proxy_handlers_batch.py
@@ -1034,6 +1034,93 @@ async def test_handle_google_batch_create_preserves_functioncall_response_order(
assert "Paris" in out[3]["parts"][0]["text"]
+async def test_handle_google_batch_create_preserves_sibling_tools(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """A batch request whose tools array carries googleSearch / codeExecution
+ alongside functionDeclarations must reach Google with those siblings intact.
+ The old code collapsed the whole array to a single functionDeclarations
+ entry, silently disabling Google Search and code execution."""
+
+ class RealConvHandler(batch_module.BatchHandlerMixin, GeminiHandlerMixin):
+ GEMINI_API_URL = "https://gemini.example"
+
+ def __init__(self) -> None:
+ self.http_client = FakeHttpClient()
+ self.metrics = FakeMetrics()
+ self.config = SimpleNamespace(
+ optimize=True, ccr_inject_tool=False, ccr_inject_system_instructions=False
+ )
+ self.openai_provider = SimpleNamespace(get_context_limit=lambda m: 8192)
+ self.openai_pipeline = SimpleNamespace(
+ apply=lambda **kw: SimpleNamespace(
+ messages=kw["messages"], timing={}, tokens_before=100, tokens_after=100
+ )
+ )
+ self.captured_body: dict | None = None
+
+ async def _next_request_id(self) -> str:
+ return "req-1"
+
+ async def _record_request_outcome(self, outcome) -> None: # noqa: ANN001
+ pass
+
+ def _extract_tags(self, headers: dict) -> dict[str, str]:
+ return {}
+
+ async def _run_compression_in_executor(self, fn, *, timeout): # noqa: ANN001, ANN201
+ return fn()
+
+ async def _store_google_batch_context(self, *a, **k) -> None: # noqa: ANN002, ANN003
+ pass
+
+ async def _retry_request(self, method, url, headers, body, **kwargs): # noqa: ANN001, ANN201
+ self.captured_body = body
+ return FakeResponse(status_code=200, content=b"{}", json_data={"name": "batches/1"})
+
+ handler = RealConvHandler()
+
+ tools = [
+ {"functionDeclarations": [{"name": "get_weather"}]},
+ {"googleSearch": {}},
+ {"codeExecution": {}},
+ ]
+ batch_body = {
+ "batch": {
+ "input_config": {
+ "requests": {
+ "requests": [
+ {
+ "request": {
+ "contents": [{"role": "user", "parts": [{"text": "hello there"}]}],
+ "tools": tools,
+ },
+ "metadata": {},
+ }
+ ]
+ }
+ }
+ }
+ }
+
+ async def payload(request): # noqa: ANN001, ANN201
+ return batch_body
+
+ monkeypatch.setattr("headroom.proxy.helpers._read_request_json", payload)
+
+ resp = await handler.handle_google_batch_create(FakeRequest("{}"), "gemini-pro")
+ assert resp.status_code == 200
+
+ out_tools = handler.captured_body["batch"]["input_config"]["requests"]["requests"][0][
+ "request"
+ ]["tools"]
+ keys = [next(iter(entry)) for entry in out_tools]
+ assert "googleSearch" in keys
+ assert "codeExecution" in keys
+ assert "functionDeclarations" in keys
+
+
@pytest.mark.asyncio
async def test_google_batch_passthrough_without_body_and_query_variants() -> None:
handler = DummyBatchHandler()