Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(telemetry): only advance usage-report baseline after a 200 (#2149)

Abhay Singh0cddac632d8a
diff --git a/CHANGELOG.md b/CHANGELOG.md
index aef7bae19..2dde03b58 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -118,6 +118,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
* **proxy:** run a cold-start fast pass before background-compression deferral so byte-identical freeze doesn't lock sessions to the uncompressed transcript. Since #1850, a session's provider-cached prefix is frozen in whatever form its cold start forwarded; deferring the WHOLE pipeline (`HEADROOM_BACKGROUND_COMPRESSION=1`, frozen=0, ≥50k tokens) therefore cached the raw transcript and forfeited the session's compression savings for its lifetime — including sub-second lossless wins like `read_lifecycle` stale-read drops, observed in the field as sessions permanently stuck at 0 savings. The deferral branch now runs the pipeline synchronously with the new `skip_kompress=True` kwarg (everything except the Kompress ML stage — the only stage that can blow the request budget per #1171) under a bounded fast-pass budget (`HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS`, default 10s), forwards the pruned form, and defers only Kompress to the background job (tagged `deferred:kompress_background`). Fail-open: on fast-pass timeout/error the request forwards uncompressed exactly as before. Units routed to Kompress under `skip_kompress` take the same fallback as when the model isn't ready.
* **init/codex:** don't overwrite the user's `hooks.json`. `_ensure_codex_hooks` wrote a fresh payload containing only Headroom's two hooks, wholesale-replacing `~/.codex/hooks.json` — so any user-managed Codex hooks (and other top-level keys) were silently destroyed on `headroom init codex`. It now read-merges: existing entries are preserved, Headroom's are deduped on the `headroom-init-codex` marker and appended, matching `_ensure_claude_hooks` / `_ensure_copilot_hooks`.
* **ccr:** detect `read_lifecycle` stale/superseded markers in the retrieve-tool injector so they stay redeemable. Those markers (`[Read content stale: … Retrieve original: hash=<hash>]`) store the original bytes in the CCR store under a valid hash, but none of `CCRToolInjector`'s patterns matched them — every pattern required the word "compressed" or the `<<ccr:` form. So on a frozen-prefix turn (both `read_lifecycle` and prefix freezing are on by default) the injector reported no compressed content, the `headroom_retrieve` tool was not injected, and the model was handed a marker advertising `Retrieve original: hash=X` with no tool to redeem it — silent data loss for stale reads, where retrieval is the only way to recover the original-at-read-time content (the exact case the #1006 guard exists to prevent). Added a pattern matching the load-bearing `Retrieve original: hash=` phrase, aligning the injector with the sibling `read_maturation` marker that was already (incidentally) detected.
+* **telemetry:** only advance the usage-report baseline after a confirmed 200. `UsageReporter._report_usage` sends usage as a delta against the last snapshot, but it called `_snapshot_metrics()` (and advanced `_last_report_time`) unconditionally after the POST — including when the send returned non-200 or raised. So a report that failed to reach the cloud (which the module is explicitly designed to tolerate) still rebased the baseline, permanently dropping that window's requests/tokens from usage-based billing/quota; the next report started from the advanced baseline and never re-included them. The baseline now advances only on a 200, so a failed send leaves the window intact for the next report to retry.
* **savings:** stop the durable savings ledger from billing free (0-priced) models at the `$3/M` fallback. `estimate_cost_usd` guarded the litellm estimate with `if priced > 0`, so a genuinely free model — where `_estimate_compression_savings_usd` correctly returns `0.0` — was treated as "unpriced" and fell through to the blended fallback rate, writing phantom cost-avoided into the JSONL ledger and surfacing it in `headroom savings`. The ledger now trusts the estimate verbatim for known models (it already falls back internally for models litellm can't price and returns `0.0` for free ones), fixing the same defect at this call site that was already fixed inside the helper.
* **init/codex:** stop `headroom init codex` from deleting per-profile provider settings. `_ensure_codex_provider` removed the root-level `model_provider`/`openai_base_url` (which init owns) with a multiline regex that matched those keys in **every** table, so a user's `[profiles.*]` overrides (e.g. `[profiles.work] model_provider = "azure"`) were silently stripped and those profiles fell through to the injected `"headroom"` default — config corruption. The strip is now scoped to the document root (everything before the first table header), so per-profile overrides are preserved while init still replaces a root-level assignment.
* **proxy:** strip output-only content blocks from request messages before forwarding. Anthropic's server-side refusal-fallback feature (`server-side-fallback-2026-06-01`) emits a `{"type":"fallback","from":{...},"to":{...}}` block inside the assistant response to signal that a refused request was re-served by the fallback model. That block is valid on the *response* path but rejected on the *request* path, so when a client replays the assistant turn the next request 400s (`invalid_request_error: messages.N.content.0: Input tag 'fallback' ...`) and the conversation gets permanently stuck through the proxy. `read_request_json_with_bytes` (Anthropic/OpenAI/Bedrock) and `_read_request_json` (Gemini) now drop such blocks — re-encoding the raw bytes so byte-faithful passthrough cannot leak the pre-strip body, backfilling a benign text block if a turn is emptied, and leaving requests without such blocks byte-identical (no cache churn).
diff --git a/headroom/telemetry/reporter.py b/headroom/telemetry/reporter.py
index eb23afd66..15217d334 100644
--- a/headroom/telemetry/reporter.py
+++ b/headroom/telemetry/reporter.py
@@ -332,15 +332,21 @@ class UsageReporter:
total_requests,
total_tokens_saved,
)
+ # Only advance the baseline after a confirmed 200. Usage is a
+ # delta against the last snapshot, so snapshotting on a failed
+ # send (non-200 or exception) would permanently drop this
+ # window's requests/tokens from billing — the next report starts
+ # from the advanced baseline and never re-includes them. Leaving
+ # both baselines intact makes the next report retry the full
+ # period. (The empty-report early-return above only advances the
+ # time, since there is nothing to lose.)
+ self._snapshot_metrics()
+ self._last_report_time = now
else:
logger.warning("Usage report returned status %d", resp.status_code)
except Exception:
logger.warning("Failed to send usage report", exc_info=True)
- # Update snapshot
- self._snapshot_metrics()
- self._last_report_time = now
-
def _snapshot_metrics(self) -> None:
"""Take a snapshot of current proxy metrics for delta computation."""
if self._proxy is None or self._proxy.cost_tracker is None:
diff --git a/tests/test_usage_reporter_snapshot.py b/tests/test_usage_reporter_snapshot.py
new file mode 100644
index 000000000..314ebded1
--- /dev/null
+++ b/tests/test_usage_reporter_snapshot.py
@@ -0,0 +1,72 @@
+"""UsageReporter must only advance its delta baseline after a confirmed 200.
+
+Snapshotting on a failed send permanently drops that window's usage from the
+delta-based usage report (billing/quota under-count)."""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import anyio
+
+from headroom.telemetry.reporter import UsageReporter
+
+
+class _Resp:
+ def __init__(self, status_code: int) -> None:
+ self.status_code = status_code
+
+ def json(self) -> dict:
+ return {}
+
+
+def _make_reporter(post_outcome) -> UsageReporter:
+ r = object.__new__(UsageReporter)
+ ct = SimpleNamespace(
+ _tokens_saved_by_model={"m": 100},
+ _tokens_sent_by_model={"m": 400},
+ _requests_by_model={"m": 5},
+ )
+ r._proxy = SimpleNamespace(cost_tracker=ct)
+ r._last_report_time = None
+ r._last_tokens_saved_by_model = {}
+ r._last_tokens_sent_by_model = {}
+ r._last_requests_by_model = {}
+ r._license_key = "k"
+ r._cloud_url = "https://cloud.example"
+ r._license_info = None
+
+ class _Client:
+ async def post(self, *args, **kwargs):
+ if isinstance(post_outcome, Exception):
+ raise post_outcome
+ return _Resp(post_outcome)
+
+ async def _get_client():
+ return _Client()
+
+ r._get_client = _get_client
+ return r
+
+
+def test_baseline_advances_only_on_success():
+ r = _make_reporter(200)
+ anyio.run(r._report_usage)
+ # Success -> baseline rebased to the current cumulative counters.
+ assert r._last_tokens_saved_by_model == {"m": 100}
+ assert r._last_requests_by_model == {"m": 5}
+
+
+def test_baseline_not_advanced_on_non_200():
+ r = _make_reporter(500)
+ anyio.run(r._report_usage)
+ # Failed send -> baseline untouched so the window is retried next report.
+ assert r._last_tokens_saved_by_model == {}
+ assert r._last_requests_by_model == {}
+
+
+def test_baseline_not_advanced_on_exception():
+ r = _make_reporter(RuntimeError("network down"))
+ anyio.run(r._report_usage)
+ assert r._last_tokens_saved_by_model == {}
+ assert r._last_requests_by_model == {}