Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(proxy/gemini): forward a non-JSON upstream body with its real status (#2174)

Abhay Singhf723925be75a
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d1713314e..af3426302 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -116,6 +116,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Bug Fixes
* **proxy:** run a cold-start fast pass before background-compression deferral so byte-identical freeze doesn't lock sessions to the uncompressed transcript. Since #1850, a session's provider-cached prefix is frozen in whatever form its cold start forwarded; deferring the WHOLE pipeline (`HEADROOM_BACKGROUND_COMPRESSION=1`, frozen=0, ≥50k tokens) therefore cached the raw transcript and forfeited the session's compression savings for its lifetime — including sub-second lossless wins like `read_lifecycle` stale-read drops, observed in the field as sessions permanently stuck at 0 savings. The deferral branch now runs the pipeline synchronously with the new `skip_kompress=True` kwarg (everything except the Kompress ML stage — the only stage that can blow the request budget per #1171) under a bounded fast-pass budget (`HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS`, default 10s), forwards the pruned form, and defers only Kompress to the background job (tagged `deferred:kompress_background`). Fail-open: on fast-pass timeout/error the request forwards uncompressed exactly as before. Units routed to Kompress under `skip_kompress` take the same fallback as when the model isn't ready.
+* **proxy/gemini:** forward a non-JSON upstream error body with its real status instead of a synthetic 502. In `handle_gemini_generate_content` the token-extraction `except (KeyError, TypeError, AttributeError)` guarding `response.json()` omitted `json.JSONDecodeError` / `ValueError`, so a non-JSON body (an HTML/empty error page from an overloaded Google/Vertex/Copilot frontend, common on 5xx/429) escaped to the outer `except Exception` and was returned as a generic 502 — discarding the real upstream `status_code` and body and defeating the client's retry/backoff. The except now catches the JSON/ValueError family, matching the all-non-text sibling branch, so the true status and body are forwarded verbatim.
* **init/codex:** don't overwrite the user's `hooks.json`. `_ensure_codex_hooks` wrote a fresh payload containing only Headroom's two hooks, wholesale-replacing `~/.codex/hooks.json` — so any user-managed Codex hooks (and other top-level keys) were silently destroyed on `headroom init codex`. It now read-merges: existing entries are preserved, Headroom's are deduped on the `headroom-init-codex` marker and appended, matching `_ensure_claude_hooks` / `_ensure_copilot_hooks`.
* **ccr:** detect `read_lifecycle` stale/superseded markers in the retrieve-tool injector so they stay redeemable. Those markers (`[Read content stale: … Retrieve original: hash=<hash>]`) store the original bytes in the CCR store under a valid hash, but none of `CCRToolInjector`'s patterns matched them — every pattern required the word "compressed" or the `<<ccr:` form. So on a frozen-prefix turn (both `read_lifecycle` and prefix freezing are on by default) the injector reported no compressed content, the `headroom_retrieve` tool was not injected, and the model was handed a marker advertising `Retrieve original: hash=X` with no tool to redeem it — silent data loss for stale reads, where retrieval is the only way to recover the original-at-read-time content (the exact case the #1006 guard exists to prevent). Added a pattern matching the load-bearing `Retrieve original: hash=` phrase, aligning the injector with the sibling `read_maturation` marker that was already (incidentally) detected.
* **install:** don't crash `headroom install status` when the health payload's `config` is a non-dict. The command did `payload.get('config', {}).get('backend', manifest.backend)`, but `dict.get(..., {})` only defaults on a *missing* key — a present-but-non-dict `config` (`null`, a string, a list, e.g. when a different or older service is answering on the port) reached the chained `.get('backend', ...)` and raised `AttributeError`, crashing the command with a raw traceback. The value is now guarded with `isinstance(config, dict)` before the lookup, mirroring `wrap.py`'s `_proxy_health_config`, so it falls back to the manifest's backend.
diff --git a/headroom/proxy/handlers/gemini.py b/headroom/proxy/handlers/gemini.py
index f9583b9a2..7d1623494 100644
--- a/headroom/proxy/handlers/gemini.py
+++ b/headroom/proxy/handlers/gemini.py
@@ -663,7 +663,15 @@ class GeminiHandlerMixin:
# Gemini returns cachedContentTokenCount for context-cached tokens
# These are charged at 10-25% of the input price depending on model
cache_read_tokens = usage.get("cachedContentTokenCount", 0)
- except (KeyError, TypeError, AttributeError) as e:
+ except (json.JSONDecodeError, ValueError, KeyError, TypeError, AttributeError) as e:
+ # A non-JSON upstream body (HTML/empty error page from an
+ # overloaded Google/Vertex frontend) makes response.json()
+ # raise JSONDecodeError (a ValueError). Without those in the
+ # tuple it escaped to the outer `except Exception` and became
+ # a synthetic 502, discarding the real upstream status/body
+ # and defeating client retry/backoff. Match the all-non-text
+ # sibling branch above, which falls through to forward the
+ # real status/content verbatim.
logger.debug(
f"[{request_id}] Failed to extract cached tokens from Gemini response: {e}"
)
diff --git a/tests/test_gemini_nonjson_status.py b/tests/test_gemini_nonjson_status.py
new file mode 100644
index 000000000..fd8673936
--- /dev/null
+++ b/tests/test_gemini_nonjson_status.py
@@ -0,0 +1,84 @@
+"""A non-JSON Gemini upstream body must not be masked as a synthetic 502."""
+
+from __future__ import annotations
+
+import json
+from types import SimpleNamespace
+
+import pytest
+
+from headroom.proxy.handlers.gemini import GeminiHandlerMixin
+
+
+class _FakeRequest:
+ def __init__(self) -> None:
+ self.headers: dict[str, str] = {}
+ self.query_params: dict[str, str] = {}
+ self.url = SimpleNamespace(path="/v1beta/models/gemini-pro:generateContent", query="")
+
+
+class _NonJsonResponse:
+ status_code = 503
+ content = b"<html>temporarily unavailable</html>"
+ headers = {"content-type": "text/html", "content-length": str(len(content))}
+
+ def json(self) -> object:
+ raise json.JSONDecodeError("not json", self.content.decode("utf-8"), 0)
+
+
+class _FakeMetrics:
+ def __init__(self) -> None:
+ self.failed: list[str] = []
+
+ async def record_failed(self, *, provider: str, model: str = "") -> None:
+ self.failed.append(f"{provider}:{model}")
+
+
+class _Handler(GeminiHandlerMixin):
+ GEMINI_API_URL = "https://gemini.example"
+
+ def __init__(self) -> None:
+ self.memory_handler = None
+ self.rate_limiter = None
+ self.usage_reporter = None
+ self.config = SimpleNamespace(
+ optimize=False,
+ anthropic_pre_upstream_memory_context_timeout_seconds=0.1,
+ )
+ self.metrics = _FakeMetrics()
+ self.outcomes = []
+
+ async def _next_request_id(self) -> str:
+ return "req-1"
+
+ async def _retry_request(self, method, url, headers, body): # noqa: ANN001, ANN201
+ return _NonJsonResponse()
+
+ async def _record_request_outcome(self, outcome) -> None: # noqa: ANN001
+ self.outcomes.append(outcome)
+
+
+async def test_generate_content_forwards_non_json_upstream_status(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ async def payload(request): # noqa: ANN001, ANN201
+ return {"contents": [{"role": "user", "parts": [{"text": "hello"}]}]}
+
+ class _Tokenizer:
+ def count_messages(self, messages): # noqa: ANN001, ANN201
+ return 7
+
+ monkeypatch.setattr("headroom.proxy.helpers._read_request_json", payload)
+ monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda model: _Tokenizer())
+
+ handler = _Handler()
+ response = await handler.handle_gemini_generate_content(_FakeRequest(), "gemini-pro")
+
+ assert response.status_code == 503
+ assert response.body == _NonJsonResponse.content
+ assert response.headers["content-type"] == "text/html"
+ assert response.headers["x-headroom-tokens-before"] == "7"
+ assert response.headers["x-headroom-tokens-after"] == "7"
+ assert handler.metrics.failed == []
+ assert handler.outcomes[0].status_code == 503