Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

perf(proxy): bound upstream calls and hot-path costs (#2852)

Tejas Chopraf624d3a00ac2
diff --git a/dashboard-cache-ttl-main.png b/dashboard-cache-ttl-main.png
new file mode 100644
index 000000000..3433b1c56
Binary files /dev/null and b/dashboard-cache-ttl-main.png differ
diff --git a/docs/context-mode-integration-analysis.md b/docs/context-mode-integration-analysis.md
new file mode 100644
index 000000000..fc21c7bf1
--- /dev/null
+++ b/docs/context-mode-integration-analysis.md
@@ -0,0 +1,378 @@
+# context-mode → Headroom: enterprise plugin & variant analysis
+
+Analysis date: 2026-07-29. Sources: `/Users/tcms/demo/context-mode` @ v1.0.169, `/Users/tcms/demo/headroom` @ main.
+
+---
+
+## 1. Bottom line
+
+context-mode and Headroom attack the same cost problem at **two different layers**, and they do not
+overlap where it matters:
+
+| | context-mode | Headroom |
+|---|---|---|
+| Interception point | agent **tool-call boundary** (host hooks + MCP) | model **API boundary** (proxy / SDK / MCP) |
+| Position relative to context | **pre-context** — data never enters | **in-context** — data already entered, gets squeezed |
+| Mechanism | admission control: block, redirect, sandbox, externalize | compression: crush, cache, retrieve |
+| Touches the wire request | never | always |
+| Loss | lossless (full content in FTS5, queryable) | lossy squeeze + hash rehydrate |
+
+Headroom's own realignment doc identifies its correct compression target as the **live zone**:
+"latest user message content + latest `tool_result` + latest `function_call_output` + latest
+`local_shell_call_output`" (`REALIGNMENT/00-overview.md`, Phase B).
+
+**That is precisely the payload context-mode intercepts one layer earlier.** Headroom Phase B is
+building a Rust engine to compress the latest tool result *after* it hits the wire. context-mode
+stops that tool result from being produced at all. These are complements, not competitors — and the
+upstream position is strictly cheaper: nothing to compress, nothing to cache-invalidate, no
+token-validation fallback needed.
+
+Three strategic unlocks, in order of value:
+
+1. **Cache safety.** Headroom's #1 identified bug class is prompt-cache busting from request
+ mutation (5 top-tier cache-killer bugs, `REALIGNMENT/00-overview.md`). context-mode has
+ *structurally zero* cache-bust risk because it never touches the request body.
+2. **Subscription safety.** The realignment flags "fingerprint-class subscription-revocation
+ risks" from `X-Headroom-*` header leakage, `anthropic-beta` mutation and re-serialization on
+ OAuth/subscription CLIs. A hook-layer product carries none of this — it is invisible to the
+ upstream. This is a *deployable-where-the-proxy-can't-go* capability.
+3. **Proxy-free deployment.** Headroom's value today requires being in the API path
+ (`127.0.0.1:8787`). Verified live this session: with the proxy down, `headroom_stats` returns all
+ zeros and `headroom_compress` no-ops. Enterprises that cannot reroute model traffic (TLS trust,
+ egress policy, subscription auth) currently get nothing. context-mode's hook+MCP model needs no
+ interposition.
+
+Zero references to context-mode exist in the Headroom tree today — clean slate.
+
+---
+
+## 2. context-mode: portable IP inventory
+
+41,617 lines of TypeScript, 11 MCP tools, 18 host adapters, npm-distributed
+(`[email protected]`, 8 runtime deps, esbuild-bundled).
+
+Ranked by *how hard it would be for Headroom to rebuild*:
+
+### Tier 1 — genuinely hard, no Headroom equivalent
+
+**1. Cross-host hook adapter layer** — `src/adapters/**` (~10K LOC), `src/adapters/types.ts`,
+`src/adapters/detect.ts` (737 lines), `configs/` (18 hosts).
+Normalizes three incompatible paradigms — `json-stdio` (Claude Code, Gemini/Qwen, Copilot, Codex,
+Kimi, Cursor, Kiro, Antigravity), `ts-plugin` (OpenCode, KiloCode, OpenClaw), `mcp-only` (Zed, Pi,
+OMP) — behind one contract: normalized `PreToolUse` / `PostToolUse` / `PreCompact` /
+`SessionStart` events, a `PlatformCapabilities` matrix, and a 5-way decision
+(`allow | deny | modify | context | ask`). Per-host install, config-format, and self-heal machinery
+included (`hooks/heal-partial-install.mjs`, `scripts/plugin-cache-integrity.mjs`).
+*Why hard to rebuild:* the value is entirely in the accumulated per-host quirks. There is no spec to
+implement against.
+
+**2. Tool-boundary policy engine** — `src/security.ts` (889 lines).
+A real policy decision point, not a regex list: glob→regex compilation, chained-command splitting
+(`&&`/`;`/`|` with escape awareness), subshell extraction, deny/ask pattern ingestion from host
+settings files, project-boundary containment (`evaluateProjectContainment` — Issue #852: an approved
+`ctx_execute_file` cannot escape the repo via a path the user couldn't see), and a
+**shell-escape scanner** (`SHELL_ESCAPE_PATTERNS`, `extractShellCommands`) that detects
+`execSync`/`subprocess`/etc. embedded inside sandboxed *non-shell* code and re-evaluates the escaped
+command against policy.
+*Why hard to rebuild:* this is the sandbox-escape prevention layer. Getting it wrong is a CVE.
+
+**3. Multi-language sandbox executor** — `src/executor.ts` (785), `src/runPool.ts`,
+`src/exit-classify.ts`, `src/truncate.ts`.
+12 languages, stdout-only egress, timeouts, background detach, output caps, exit classification.
+Enforces the "Think in Code" contract: the agent programs the analysis, only the answer enters
+context.
+
+**4. Lossless externalization store** — `src/store.ts` (2,071 lines).
+Dual SQLite FTS5 index — a tokenized `chunks` table *plus* a `chunks_trigram` table for
+substring/identifier search where BM25 tokenization fails on code — with a `vocabulary` table and
+schema migration path. Auto-externalizes any output >100 KB into FTS5 and returns a pointer.
+Nothing is discarded; the model queries on demand.
+
+### Tier 2 — valuable, but partially duplicated in Headroom
+
+**5. Counterfactual savings accounting** — `src/session/analytics.ts` (3,085 lines),
+`src/session/project-attribution.ts`, `src/session/db.ts` (1,726).
+`ContextSavings`, `ThinkInCodeComparison`, `RealBytesStats`, `MultiAdapterLifetimeStats`,
+`enumerateAdapterDirs()`. Measures *what would have entered context but didn't* — a different and
+harder quantity than Headroom's `savings_ledger.py`, which records actual compression deltas.
+Session event ledger + `tool_calls` + resume + per-project attribution.
+
+**6. Multi-vendor pricing catalog** — `src/session/pricing.ts` + `model-prices.json`.
+61 curated models × 4 rate buckets (input / output / cache-read / cache-write), refreshed from
+litellm, unknown model → `null` rather than a silently wrong Claude rate.
+**Overlaps `headroom/pricing/*` heavily. Do not port.**
+
+### Tier 3 — do not port
+
+Compression heuristics, memory/graph/relevance, telemetry transport, dashboard, install UX,
+update-check. Headroom has all of these, more mature, and Phase B/H is actively consolidating them.
+
+---
+
+## 3. Headroom's actual extension seams
+
+Verified entry-point groups (all `importlib.metadata`-discovered, all opt-in):
+
+| Seam | Group | Contract | Source |
+|---|---|---|---|
+| Proxy extension | `headroom.proxy_extension` | `install(app: FastAPI, config: ProxyConfig) -> None` | `headroom/proxy/extensions.py:52` |
+| Pipeline extension | `headroom.pipeline_extension` | `on_pipeline_event(PipelineEvent) -> PipelineEvent \| None` over 11 stages | `headroom/pipeline.py:13,68` |
+| Learn plugin | `headroom.learn_plugin` | — | `headroom/learn/registry.py:44` |
+| Memory text store | `headroom.memory_text` | — | `headroom/memory/config.py:41`, `factory.py:57` |
+| Memory vector store | `headroom.memory_vector` | — | `headroom/memory/config.py:34` |
+| Memory store | `headroom.memory_store` | — | `headroom/memory/config.py:25` |
+| CCR backend | `headroom.ccr_backend` | — | `headroom/cache/compression_store.py:981` |
+| Compression hooks | (subclass, not entry point) | `pre_compress` / `compute_biases` / `post_compress` | `headroom/hooks.py:1-31` |
+
+Two things worth noting:
+
+- `headroom/proxy/extensions.py:32` states an explicit **stability contract**: changing
+ `install(app, config)` or the group name requires a deprecation cycle. This is a supported public
+ seam, not an accident.
+- `headroom/hooks.py:16` says outright: *"Headroom SaaS implements position-aware compression and
+ cross-turn deduplication via these hooks."* The open-core split is already designed in.
+
+**The exemplar to copy:** `plugins/headroom-oauth2/` — own `pyproject.toml`, own `LICENSE`, own
+`SPEC.md`, registers on `headroom.proxy_extension`, dormant until `--proxy-extension oauth2`,
+all config via env, "zero core changes." That is the enterprise plugin template.
+
+**The precedent to copy:** `headroom/lean_ctx/installer.py` and `headroom/rtk/installer.py` —
+Headroom already ships thin installers that adopt sibling products. `plugins/headroom-agent-hooks`
+already installs startup hooks into Claude Code and Copilot CLI. The socket exists.
+
+**The gap:** Headroom has *no tool-boundary interception anywhere*. It sees `tool_use`/`tool_result`
+only as message content after the fact (`headroom/parser.py`, `headroom/tokenizers/*`). Its
+`PipelineStage` enum has no tool-result stage. Everything context-mode does is upstream of
+Headroom's earliest hook.
+
+---
+
+## 4. Proposed plugins & variants
+
+Ranked by value ÷ effort.
+
+### P1 — `headroom-recall`: FTS5+trigram lossless store as `headroom.memory_text`
+
+**What:** port `src/store.ts` behind the existing `headroom.memory_text` seam.
+
+**Why this first:** it is the smallest diff onto an *already-existing* contract, and it fixes a real
+product limitation. Today `headroom_retrieve(hash)` requires you to *know the hash* — the tool
+description literally says "hash comes from compression markers like `[N items compressed... hash=abc123]`".
+With an FTS5-backed store you get `retrieve-by-query`: "what did that build log say about OOM"
+instead of "paste hash abc123". The trigram index matters specifically because BM25 tokenization
+loses identifiers and stack frames.
+
+Composes rather than replaces: `compress` → return squeezed text + hash → store the *original* in
+FTS5 → rehydrate by hash **or** by query. Also a natural `headroom.ccr_backend` implementation —
+the realignment wants "CCR hardens: persistent backend" (Phase B), and this is one.
+
+**Enterprise variant:** shared team store, retention/TTL policy, per-project scoping (context-mode
+already has `project-attribution.ts`), audit of every retrieval.
+
+**Effort:** medium. Reimplement in Python/Rust against Headroom's memory interface, or ship the
+node store as a sidecar. Do not port the MCP tool surface — only the store.
+
+### P2 — `headroom-admission`: tool-boundary admission control across 18 hosts
+
+**What:** context-mode's adapter + hook layer, distributed the way `plugins/openclaw` and
+`plugins/opencode` already are (TS package under `plugins/`), reporting savings into Headroom's
+`savings_ledger.py` JSONL and emitting Headroom pipeline events.
+
+**Why:** this is the strategic piece. It gives Headroom:
+- a **pre-wire** enforcement point, upstream of Phase B's live-zone engine, with no cache-bust and
+ no token-validation fallback required;
+- coverage of **18 agent hosts** — the realignment's Phase G wants to "extend wrap CLIs (cline,
+ continue, goose, openhands)"; this is that work already done, and then some;
+- a deployment mode that works under **subscription auth**, where the proxy is a revocation risk.
+
+**Enterprise value — this is the DLP story Headroom cannot currently tell.** A `curl` inside a Bash
+tool call never touches the proxy, so Headroom is blind to it. context-mode blocks
+`curl`/`wget`/`WebFetch`/inline `fetch()`/`requests.get` at the tool boundary and forces network
+egress through `ctx_fetch_and_index`. That converts a token-savings feature into an
+**egress-control** feature — a different budget line and a different buyer.
+
+**Effort:** high, but it's mostly packaging + a reporting bridge, not a rewrite. Keep it TypeScript;
+Phase H retires Python *proxy* code but explicitly preserves "CLI wrappers, RTK installer" — the
+installer layer is the surviving Python, and it can shell out.
+
+### P3 — `headroom-policy` (Enterprise, license-gated): the PDP
+
+**What:** `src/security.ts` as a policy decision point, plus centrally-managed org rulesets.
+
+Two attach points: the hook layer from P2 (tool-level `allow/deny/ask`), and
+`headroom.pipeline_extension` at `PRE_SEND` (prompt-level policy). Feeds `headroom/audit/`.
+
+**Enterprise features that only make sense paid:** central policy service, org-wide allow/deny
+rulesets, project-boundary containment enforcement, shell-escape detection inside sandboxed code,
+tamper-evident audit trail, per-team reporting. Gate it with the ELv2 license key (see §6).
+
+**Effort:** medium. The engine exists and is tested (`tests/security/`, `src/security.ts` 889 lines);
+the work is the control plane.
+
+### P4 — `headroom-sandbox`: Think-in-Code execution
+
+**What:** `executor.ts` exposed as a Headroom MCP tool (`headroom_execute`), 12 languages,
+stdout-only.
+
+**Why:** this is the mechanism behind context-mode's largest measured savings —
+`ctx_execute_file` returns 98% savings across 315 KB of real fixtures (`BENCHMARK.md` Part 1),
+versus 82% for index+search (Part 2). Programming the analysis beats compressing the output.
+
+Must ship *with* P3: the shell-escape scanner is what stops the sandbox being an escape hatch.
+
+**Effort:** medium-high. Runtime isolation is the hard part; `headroom` already has a `sandbox` extra
+in `pyproject.toml` to build on.
+
+### P5 — `headroom-attribution`: counterfactual savings + per-project cost
+
+**What:** port the *methodology* from `session/analytics.ts` — `RealBytesStats`,
+`ThinkInCodeComparison`, `enumerateAdapterDirs`, `project-attribution.ts` — into Headroom's
+`savings_ledger` / `reporting` / `dashboard`.
+
+**Why:** Headroom measures compression deltas (what it squeezed). context-mode measures the
+counterfactual (what never entered). Enterprise buyers want the second number, sliced by team and
+repo. Do **not** port `pricing.ts` — `headroom/pricing/*` already does this with litellm resolution.
+
+**Merge, don't port.** `headroom/audit/reads.py` is already a counterfactual measurement tool over
+the same Claude Code transcript corpus (see §8). It has the better mechanism taxonomy — identical
+repeat, subset containment, write-readback, stale, line-number scaffolding, context residency,
+cache-death windows. `analytics.ts` has the multi-host coverage and per-project attribution it
+lacks. Combine the two rather than adding a third implementation.
+
+**Effort:** low-medium, mostly a metrics-definition merge.
+
+### Variants (packaging, not code)
+
+- **Headroom No-Proxy Edition** — P1+P2 only, zero API interposition. Sells to buyers who cannot
+ reroute model traffic and to every subscription-auth user. Removes the single biggest deployment
+ blocker Headroom has.
+- **Headroom Admission Control (Enterprise)** — P2+P3+P4 with a central policy plane and fleet
+ enrollment across 18 hosts. Positioned as AI-agent DLP/governance, not token savings.
+- **Headroom Fleet** — P5 + `enumerateAdapterDirs` for org-wide rollout state and cost reporting.
+
+---
+
+## 5. Evidence base
+
+context-mode's `BENCHMARK.md`: 21 scenarios, 376 KB raw → 16.5 KB context, **96% overall**, all
+fixtures captured from real tool invocations (Context7, Playwright, `gh`, vitest, tsc, nginx logs,
+`git log`, analytics CSV) rather than synthetic. Honest about its weak cases — 13% on a 0.4 KB
+Playwright network dump, and Part 2 openly explains why index+search only reaches 50-93% (it returns
+exact code blocks rather than summaries, by design).
+
+Test suite: 125 tests across executor/store/MCP-integration/ecosystem, plus 45 test dirs in `tests/`
+covering adapters, security, session, hooks, analytics.
+
+That's a defensible enough evidence base to reuse in Headroom's own materials, and the fixture corpus
+itself is reusable for Headroom's `benchmarks/`.
+
+---
+
+## 6. Blockers — resolve these before writing code
+
+**1. License incompatibility (hard blocker).**
+context-mode is **Elastic License 2.0**, "Copyright 2026 Mert Koseoglu". Headroom is
+**Apache-2.0**, "Copyright 2025 Headroom Contributors".
+
+- ELv2 code **cannot** be merged into the Apache-2.0 core. Not a technicality — it would relicense
+ Headroom's core.
+- ELv2 forbids providing the software "to third parties as a hosted or managed service." That
+ directly constrains `headroom-managed/`.
+- Different copyright holders means this needs an **IP arrangement between entities**, not an
+ engineering decision.
+
+The good news: Headroom's plugin architecture is exactly the boundary that makes this tractable.
+A separate package with its own `pyproject.toml` and its own `LICENSE`, registered on an entry
+point — the `plugins/headroom-oauth2/` shape — can carry ELv2 while core stays Apache-2.0. ELv2 is
+also the *right* license for a license-key-gated enterprise tier; it explicitly contemplates one.
+
+Recommendation: any context-mode-derived code ships as separately-licensed plugin packages under
+`plugins/`, never vendored into `headroom/`. Get the IP arrangement in writing first.
+
+**2. Realignment collision.**
+Phases A–I are ~40 PRs / 8–13 weeks and include deleting ~25K LOC. Do not open a new integration
+front mid-Phase-B. P1 (`headroom.memory_text` / `ccr_backend`) is the exception — it *serves* Phase
+B's "CCR hardens: persistent backend" goal rather than competing with it.
+
+**3. Phase H direction.**
+Python proxy code is being retired. Write nothing new in `headroom/proxy/`. Target the surviving
+layers: installers, memory writers, CLI wrappers, and Rust.
+
+---
+
+## 7. Sequencing
+
+| Order | Item | Gate |
+|---|---|---|
+| 0 | IP/licensing arrangement | before any code |
+| 1 | P1 `headroom-recall` — FTS5 store on `memory_text`/`ccr_backend` | lands inside Phase B, serves it |
+| 2 | P2 `headroom-admission` — 18-host hook layer under `plugins/` | after Phase A stabilizes |
+| 3 | Variant: **No-Proxy Edition** = P1+P2 | as soon as P2 works on 3+ hosts |
+| 4 | P3 `headroom-policy` (Enterprise, ELv2, key-gated) | after P2 |
+| 5 | P4 `headroom-sandbox` | with P3, never before |
+| 6 | P5 `headroom-attribution` | opportunistic |
+
+---
+
+## 8. Follow-up verification
+
+All four items flagged as open in the first pass are now resolved.
+
+**`headroom-managed/` is the SaaS arm, and it is unlicensed.**
+`headroom-managed/pyproject.toml`: `name = "headroom-managed"`, `description = "Headroom SaaS
+Platform - Managed context window optimization"`, `version = 0.1.0`. It has `app/auth.py`,
+`app/middleware/`, `app/routes/`, `app/services/`, `app/models.py`, alembic migrations, and a
+`pilot/`. There is **no `license` field and no LICENSE file** — i.e. proprietary by default.
+
+This *sharpens* the §6 blocker rather than easing it. ELv2 forbids providing the software "to third
+parties as a hosted or managed service." The product whose name is literally *Managed* is the one
+place context-mode-derived code cannot go without an explicit commercial grant from the copyright
+holder. Plan the plugin boundary so that `headroom-managed` consumes only Apache-2.0 core
+interfaces, never ELv2 implementations.
+
+**`headroom/audit/reads.py` does not overlap P3 — and it independently validates the whole thesis.**
+It is a *measurement* tool, not an audit trail: it streams Claude Code `*.jsonl` transcripts to size
+"the addressable bytes for each Read compression mechanism... so defaults are set from traffic, not
+theory." No policy, no tamper-evidence. P3's audit trail remains a gap.
+
+Two lines in its docstring are the most useful corroboration in either repo:
+
+- *"context residency — how many assistant turns each Read stays in context (the multiplier on its
+ prefix-cache read cost; **the case for compress-before-cache-entry**)"* — Headroom is already
+ arguing, from its own traffic, for moving earlier in the pipeline. context-mode is the terminus of
+ that argument: compress before **context** entry, not merely before cache entry.
+- *"identical repeat — a dedup mechanism for this was prototyped and removed: it measured 0.1% of
+ Read bytes on real traffic."* — Headroom has already empirically established that
+ message-history-level dedup is worthless. The addressable bytes are at the tool boundary, not in
+ history. That is the same conclusion the realignment reached from the cache side, arrived at
+ independently from the traffic side.
+
+It *does* overlap **P5** — `audit/reads.py` and context-mode's `session/analytics.ts` are two
+independent implementations of counterfactual measurement over the same transcript corpus. Merge
+them rather than porting; `audit/reads.py` has the better mechanism taxonomy, `analytics.ts` has
+multi-host coverage and per-project attribution.
+
+**No plugin-authoring docs exist.** `docs/` is a Next.js site (`app/`, `content/`, `components/`);
+`wiki/` has nothing on extension authoring (only `macos-deployment.md` matched). `plugins/headroom-oauth2/SPEC.md`
+remains the de-facto authoring reference — which means whichever plugin lands first sets the house
+style. Worth writing the authoring doc as part of P1.
+
+**Headroom publishes no benchmark results.** `benchmarks/` is 29 runner scripts with no committed
+results artifacts, so no like-for-like number exists to compare against context-mode's 96%. The
+comparison has to be run. The harness is there and is unusually strong on exactly the axis that
+matters: `prefix_cache_benchmark.py`, `cache_bust_trace_report.py`, `cache_validation_bundle.py`,
+`synthetic_token_cache_bust_report.py`, `proxy_mode_benchmark.py`, `agent_cost_benchmark.py`,
+`real_world_agent_benchmark.py`. Use it to *prove* the §1 cache-safety claim empirically rather than
+asserting it — a measured "zero cache-bust events" result is the strongest possible artifact for the
+No-Proxy Edition.
+
+**Bonus finding — the platform axes are orthogonal.**
+`docs/platform-feature-matrix.json` (schema v1, updated 2026-07-06) tracks coverage across
+`["linux", "macos", "windows"]` — Headroom's platform axis is **operating system**. context-mode's
+platform axis is **agent host** (18 of them). Headroom tracks no host-coverage matrix at all. P2
+therefore fills a dimension that does not currently exist in Headroom's own feature accounting,
+which also means it needs a second matrix rather than new rows in this one.
+
+*Process note:* six subagents were dispatched across this analysis and all six stalled at the
+600-second watchdog; one reported "Bash is temporarily unavailable" before dying, so the failures
+were tool-layer, not analytical. Every finding in this document was verified directly.
diff --git a/headroom/backends/litellm.py b/headroom/backends/litellm.py
index 8430789ce..60fedfc3a 100644
--- a/headroom/backends/litellm.py
+++ b/headroom/backends/litellm.py
@@ -413,6 +413,40 @@ PROVIDER_REGISTRY: dict[str, ProviderConfig] = {
}
+# How long an upstream call may go silent before we give up on it.
+#
+# WHY THIS EXISTS. There was no timeout here at all, so a request the upstream
+# never answered blocked its caller forever. Observed 2026-08-07 under load:
+# four agent workers sat on ESTABLISHED connections for 36+ minutes while this
+# proxy answered /readyz in 0.11s. No error, no retry, no log line -- the
+# client just stops. That is the worst shape a failure can take, because it is
+# indistinguishable from slow work and no supervisor can tell the difference.
+#
+# A float, not an httpx.Timeout, on purpose: litellm expands a float into all
+# four httpx phases, so for a STREAMING call this becomes the maximum gap
+# BETWEEN CHUNKS rather than a cap on total generation time. A long answer
+# streaming steadily is never cut off; a stalled one dies. That is the
+# semantic we want, and it falls out of the simpler type.
+#
+# 600s is deliberately generous -- long enough that no healthy call is at
+# risk, short enough that a hang surfaces within a coffee break instead of
+# never.
+UPSTREAM_TIMEOUT_ENV = "HEADROOM_UPSTREAM_TIMEOUT"
+DEFAULT_UPSTREAM_TIMEOUT = 600.0
+
+
+def _upstream_timeout() -> float:
+ """Seconds. Never raises; a junk env value must not disable the timeout."""
+ import os
+
+ try:
+ v = float(os.getenv(UPSTREAM_TIMEOUT_ENV, DEFAULT_UPSTREAM_TIMEOUT))
+ except (TypeError, ValueError):
+ return DEFAULT_UPSTREAM_TIMEOUT
+ # 0 or negative would mean "no timeout" to httpx, which is the bug.
+ return v if v > 0 else DEFAULT_UPSTREAM_TIMEOUT
+
+
def get_provider_config(provider: str) -> ProviderConfig:
"""Get provider config, with fallback for unknown providers."""
if provider in PROVIDER_REGISTRY:
@@ -925,6 +959,9 @@ class LiteLLMBackend(Backend):
logger.debug(f"LiteLLM request: model={litellm_model}")
# Make the call
+ # Bounded, always: an upstream that never answers must not
+ # block the caller forever. setdefault so an explicit value wins.
+ kwargs.setdefault("timeout", _upstream_timeout())
response = await acompletion(**kwargs)
# Convert to Anthropic format
@@ -1055,6 +1092,9 @@ class LiteLLMBackend(Backend):
kwargs["stream_options"] = {"include_usage": True}
# Stream content — blocks emitted dynamically based on response
+ # Bounded, always: an upstream that never answers must not
+ # block the caller forever. setdefault so an explicit value wins.
+ kwargs.setdefault("timeout", _upstream_timeout())
response = await acompletion(**kwargs)
output_tokens = 0
current_block_index = -1
@@ -1283,6 +1323,9 @@ class LiteLLMBackend(Backend):
logger.debug(f"LiteLLM OpenAI request: model={litellm_model}")
# Make the call
+ # Bounded, always: an upstream that never answers must not
+ # block the caller forever. setdefault so an explicit value wins.
+ kwargs.setdefault("timeout", _upstream_timeout())
response = await acompletion(**kwargs)
# Build the usage block. LiteLLM normalizes prompt-cache stats from
@@ -1452,6 +1495,9 @@ class LiteLLMBackend(Backend):
elif headers.get("x-api-key"):
kwargs["api_key"] = headers["x-api-key"]
+ # Bounded, always: an upstream that never answers must not
+ # block the caller forever. setdefault so an explicit value wins.
+ kwargs.setdefault("timeout", _upstream_timeout())
response = await acompletion(**kwargs)
async for chunk in response:
diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.py
index 26e9f68ee..8251b3590 100644
--- a/headroom/proxy/handlers/anthropic.py
+++ b/headroom/proxy/handlers/anthropic.py
@@ -2562,8 +2562,14 @@ class AnthropicHandlerMixin:
# turn-hook fold (optimized_messages is post-hook). Runs unconditionally.
try:
_orig_snapshot = original_client_messages # noqa: F821 (bound at request start)
- original_tokens = tokenizer.count_messages(_orig_snapshot)
- optimized_tokens = tokenizer.count_messages(optimized_messages)
+ # Off the event loop (#2810): both passes are CPU-bound real BPE
+ # since #2543, and running them inline stalled every other
+ # in-flight request on the same process (~1s on a 2.3 MB body).
+ # Same tokenizer instance, so the reported values are unchanged.
+ original_tokens = await asyncio.to_thread(tokenizer.count_messages, _orig_snapshot)
+ optimized_tokens = await asyncio.to_thread(
+ tokenizer.count_messages, optimized_messages
+ )
# Fold the tool-schema/desc compaction delta into BOTH endpoints so
# tok_before - tok_after == tok_saved stays coherent in the PERF line
# (count_messages never sees tool bytes). Same shape as the OpenAI chat
@@ -2657,7 +2663,11 @@ class AnthropicHandlerMixin:
parsed_original = json.loads(original_body_bytes)
if parsed_original != body:
body_mutation_tracker.mark_mutated("structural_diff_vs_original")
- except (json.JSONDecodeError, ValueError):
+ # MemoryError is not a ValueError, so a re-parse spike on 1M-context
+ # bodies used to escape and abort an otherwise-fine request (#2768).
+ # This block is a safety net; marking mutated is already the safe
+ # outcome (it forces canonical re-serialization).
+ except (json.JSONDecodeError, ValueError, MemoryError, RecursionError):
body_mutation_tracker.mark_mutated("original_unparseable")
if (
diff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.py
index a7d22449b..23751c65e 100644
--- a/headroom/proxy/handlers/openai.py
+++ b/headroom/proxy/handlers/openai.py
@@ -3721,8 +3721,11 @@ class OpenAIHandlerMixin:
# (result.tokens_before), which mismatches optimized_tokens (provider tokenizer)
# and yields impossible tok_after>tok_before. Recount original from the
# pre-compression snapshot so the message delta is on one scale.
- original_tokens = tokenizer.count_messages(original_client_messages)
- optimized_tokens = tokenizer.count_messages(body["messages"])
+ # Off the event loop (#2810): see the matching note in the Anthropic handler.
+ original_tokens = await asyncio.to_thread(
+ tokenizer.count_messages, original_client_messages
+ )
+ optimized_tokens = await asyncio.to_thread(tokenizer.count_messages, body["messages"])
if tool_tokens_before_compaction > 0:
try:
tool_tokens_after_compaction = tokenizer.count_text(_json_debug_dumps(tools))
diff --git a/headroom/proxy/prometheus_metrics.py b/headroom/proxy/prometheus_metrics.py
index fe477290b..889ed60be 100644
--- a/headroom/proxy/prometheus_metrics.py
+++ b/headroom/proxy/prometheus_metrics.py
@@ -417,14 +417,14 @@ class PrometheusMetrics:
return total_input_tokens, total_input_cost_usd
try:
- cost_stats = self.cost_tracker.stats()
+ # totals() rather than stats(): identical numbers, without the
+ # 31-day cost-record walk that stats()["budget_basis"] performs and
+ # this caller throws away. See CostTracker.totals.
+ tracked_input_tokens, tracked_input_cost_usd = self.cost_tracker.totals()
except Exception:
logger.debug("Failed to read cost tracker totals for savings history", exc_info=True)
return total_input_tokens, total_input_cost_usd
- tracked_input_tokens = cost_stats.get("total_input_tokens")
- tracked_input_cost_usd = cost_stats.get("total_input_cost_usd")
-
if tracked_input_tokens is not None:
try:
total_input_tokens = self._savings_tracker_input_tokens_offset + max(
diff --git a/run-all-plugins.sh b/run-all-plugins.sh
new file mode 100755
index 000000000..f8b9b2ec9
--- /dev/null
+++ b/run-all-plugins.sh
@@ -0,0 +1,171 @@
+#!/usr/bin/env bash
+# ─────────────────────────────────────────────────────────────────────────────
+# run-all-plugins.sh — install + configure + run the Headroom proxy with ALL 5
+# enterprise plugins, the coding savings-profile, and ML compression offloaded
+# to the Kompress-v2 Modal endpoint. Then confirm everything loaded.
+#
+# Plugins : lossless_guard, skill_search, observability, tier_router, tool_search
+# Extra : headroom-ai[sandbox] (torch-free proxy; ML offloaded to Modal)
+# Profile : coding (HEADROOM_SAVINGS_PROFILE) + cache mode (prefix-cache safe)
+#
+# Secrets are SOURCED from ~/env.txt and ~/.headroom/plugins.env — never inlined.
+# Re-runnable: install is skipped when already satisfied (FORCE_INSTALL=1 forces).
+# ─────────────────────────────────────────────────────────────────────────────
+set -euo pipefail
+
+HR=/Users/tcms/demo/headroom
+VENV="$HR/.venv"
+PORT="${HEADROOM_PORT:-8787}"
+ENV_TXT="${ENV_TXT:-$HOME/env.txt}"
+PLUGINS_ENV="$HOME/.headroom/plugins.env"
+LOG="${HEADROOM_LOG:-$HOME/.headroom/logs/proxy-all-plugins.log}"
+mkdir -p "$(dirname "$LOG")"
+
+# ── 1. venv ──────────────────────────────────────────────────────────────────
+# `python`/`pip`/`uv` are broken system-wide on this box — always use the venv,
+# and `python -m pip` (the .venv/bin/pip shim is broken too).
+# shellcheck disable=SC1091
+source "$VENV/bin/activate"
+PY="$VENV/bin/python"
+
+# ── 2. install (guarded) ──────────────────────────────────────────────────────
+# headroom-ai[sandbox] pulls proxy,code,relevance,reports,otel,html,mcp,spreadsheet
+# (all torch-free — heavy ML is offloaded to the Modal Kompress endpoint below).
+# The 5 plugins install --no-deps so pip won't drag PyPI's headroom-ai over the
+# local editable one; headroom-license is their shared Ed25519 verifier.
+need_install=1
+if [ "${FORCE_INSTALL:-0}" != "1" ]; then
+ n=$("$PY" -c 'import opentelemetry; from headroom.proxy.extensions import discover; print(len(list(discover())))' 2>/dev/null || echo 0)
+ [ "$n" = "5" ] && need_install=0
+fi
+if [ "$need_install" = "1" ]; then
+ avail=$(df -g "$HR" 2>/dev/null | awk 'NR==2{print $4}')
+ echo "▶ disk: ${avail:-?}Gi free before install"
+ if [ -n "$avail" ] && [ "$avail" -lt 2 ]; then
+ echo "!! <2Gi free — aborting before heavy install (free space, then re-run)"; exit 1
+ fi
+ echo "▶ installing pip+maturin, then headroom-ai[sandbox] + license + 5 plugins (editable)…"
+ "$PY" -m pip install -U pip maturin
+ # litellm >=1.92 ships an sdist-only Rust bridge whose AWS-SDK crates need rustc>=1.94.1;
+ # the default rustup toolchain here is older (pip builds litellm in a temp dir that misses
+ # the repo's 1.95 pin), so pin to the last pure-Python wheel line (1.91.4). Satisfies
+ # headroom's litellm>=1.86.2,<2.0 and skips the Rust build entirely.
+ "$PY" -m pip install "litellm<1.92"
+ "$PY" -m pip install -e "${HR}[sandbox]" "litellm<1.92"
+ "$PY" -m pip install -e /Users/tcms/demo/headroom-license
+ for p in lossless-guard skill-search observability tier-router tool-search; do
+ "$PY" -m pip install -e "/Users/tcms/demo/headroom-${p}" --no-deps
+ done
+else
+ echo "▶ install satisfied (5 extensions discovered) — skipping (FORCE_INSTALL=1 to force)"
+fi
+
+# ── 3. secrets from ~/env.txt ─────────────────────────────────────────────────
+# Provides: OPENAI_API_KEY, ANTHROPIC_API_KEY, FIREWORKS_API_KEY (upstream creds);
+# LANGFUSE_{PUBLIC,SECRET}_KEY + LANGFUSE_BASE_URL (observability sink);
+# HEADROOM_KOMPRESS_ENDPOINT + _TOKEN (Modal ML offload).
+[ -f "$ENV_TXT" ] || { echo "!! $ENV_TXT not found"; exit 1; }
+set -a; # shellcheck disable=SC1090
+source "$ENV_TXT"; set +a
+
+# ── 4. plugin license (Ed25519, offline, wildcard) ────────────────────────────
+# HEADROOM_LICENSE + HEADROOM_LICENSE_PUBKEY. This is SEPARATE from the OSS cloud
+# key (HEADROOM_LICENSE_KEY) — the banner will still say "OSS (no license key)",
+# but each plugin prints "license accepted". Fallback: skip verification entirely.
+if [ -f "$PLUGINS_ENV" ]; then
+ set -a; # shellcheck disable=SC1090
+ source "$PLUGINS_ENV"; set +a
+else
+ echo "▶ $PLUGINS_ENV missing — using dev license bypass"
+ export HEADROOM_LICENSE_DEV=1
+fi
+
+# ── 5. Kompress ML offload → Modal ────────────────────────────────────────────
+# Setting HEADROOM_KOMPRESS_ENDPOINT (+_TOKEN) alone routes Kompress inference to
+# the Modal endpoint (content_router._get_kompress_remote). No other flag needed;
+# HEADROOM_COMPRESS_ALLOW_REMOTE is a different thing (remote upstreams, not this).
+: "${HEADROOM_KOMPRESS_ENDPOINT:?must be set in $ENV_TXT}"
+export HEADROOM_KOMPRESS_ENDPOINT_TOKEN="${HEADROOM_KOMPRESS_ENDPOINT_TOKEN:-}"
+
+# ── 6. observability sink → Langfuse + spend attribution ──────────────────────
+# HEADROOM_LANGFUSE_ENABLED must be explicitly truthy (LANGFUSE_* creds come from
+# env.txt). Traces (agent.turn / llm.turn spans with gen_ai.usage.cost) land in
+# Langfuse. Spend is opt-in: HEADROOM_MODEL_PRICES is {model-substr:{in,out}} in
+# USD per 1K tokens. (Per-request identity — org/team/user/session — is supplied
+# by the CLIENT via x-headroom-* headers, not settable here.)
+export HEADROOM_LANGFUSE_ENABLED=1
+export HEADROOM_LANGFUSE_SERVICE_NAME=headroom-proxy
+export HEADROOM_MODEL_PRICES='{"claude-opus":{"in":0.015,"out":0.075},"claude-sonnet":{"in":0.003,"out":0.015},"gpt-5":{"in":0.00125,"out":0.01},"gpt-4":{"in":0.003,"out":0.012}}'
+# Metrics (counters) need a separate OTLP endpoint — none in env.txt, so left off:
+# export HEADROOM_OTEL_METRICS_ENABLED=1 HEADROOM_OTEL_METRICS_ENDPOINT=http://localhost:4318
+
+# ── 7. tier_router ─────────────────────────────────────────────────────────────
+# Only stamps service_tier on the wire (no token delta). OpenAI 'flex' is only
+# auto-selected for models declared eligible here. Anthropic tiers are a no-op by
+# default. Clients force a tier with x-headroom-tier / x-headroom-background: 1.
+export HEADROOM_TIER_FLEX_MODELS="${HEADROOM_TIER_FLEX_MODELS:-gpt-5,gpt-4.1,o4-mini}"
+
+# ── 8. plugin tuning (defaults shown; override as needed) ─────────────────────
+# skill_search fires on Anthropic w/ >=min skills; tool_search on synthetic-tier
+# providers w/ >=min tools; lossless_guard lossy tier is opt-in (kept OFF).
+export HEADROOM_SKILL_SEARCH_MIN_SKILLS="${HEADROOM_SKILL_SEARCH_MIN_SKILLS:-8}"
+export HEADROOM_TOOL_SEARCH_MIN_TOOLS="${HEADROOM_TOOL_SEARCH_MIN_TOOLS:-5}"
+# export HEADROOM_LOSSLESS_GUARD_LOSSY=1 # opt-in irreversible Bash-noise drop
+
+# ── 9. coding profile + mode ───────────────────────────────────────────────────
+# savings_profile=coding tunes the pipeline for coding-agent traffic; cache mode
+# freezes prior turns to preserve the provider prefix-cache (what coding wants).
+export HEADROOM_SAVINGS_PROFILE=coding
+
+# ── 10. run + confirm ──────────────────────────────────────────────────────────
+cleanup() { [ -n "${PROXY_PID:-}" ] && kill "$PROXY_PID" 2>/dev/null || true; }
+trap cleanup INT TERM EXIT
+
+echo "▶ starting proxy on :$PORT (profile=coding, mode=cache, all 5 extensions)…"
+headroom proxy --port "$PORT" --mode cache --proxy-extension '*' > "$LOG" 2>&1 &
+PROXY_PID=$!
+
+# wait for readiness (no foreground sleep on this harness)
+curl -s --retry 40 --retry-delay 1 --retry-all-errors --max-time 60 \
+ "http://127.0.0.1:$PORT/health" >/dev/null 2>&1 || true
+
+echo
+echo "══════════════════ CONFIRMATION ══════════════════"
+echo "── extensions loaded (from $LOG) ──"
+grep -iE "Extensions:|license accepted|installed \(" "$LOG" | sed 's/^/ /' || true
+
+echo "── Modal Kompress endpoint reachable? ──"
+code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "$HEADROOM_KOMPRESS_ENDPOINT" || echo "unreachable")
+echo " $HEADROOM_KOMPRESS_ENDPOINT -> HTTP $code (any response = up; offload runs on real traffic)"
+
+echo "── /stats surfaces (empty until traffic flows) ──"
+curl -s --max-time 5 "http://127.0.0.1:$PORT/stats" | "$PY" -c '
+import sys,json
+d=json.load(sys.stdin)
+print(" extension_savings :", d.get("extension_savings"))
+print(" by_layer :", list(d.get("savings",{}).get("by_layer",{})))
+print(" tokens_saved_by_strat:", d.get("tokens_saved_by_strategy"))
+print(" otel.enabled :", d.get("otel",{}).get("enabled"))
+print(" langfuse.enabled :", d.get("langfuse",{}).get("enabled"))
+' 2>/dev/null || echo " (stats not ready)"
+
+cat <<EOF
+
+── where each effect shows up ──
+ lossless_guard -> dashboard (compression layer) + /stats.tokens_saved_by_strategy
+ skill_search -> /stats.extension_savings (NOT dashboard) — Anthropic client, >=8 skills
+ tool_search -> /stats.extension_savings (NOT dashboard) — synthetic-tier client, >=5 tools
+ observability -> Langfuse UI (spans + gen_ai.usage.cost) — send x-headroom-org/user/session
+ tier_router -> service_tier on the wire / provider bill (no token delta)
+
+── drive traffic (two clients — they exercise different plugins) ──
+ Claude Code : ANTHROPIC_BASE_URL=http://localhost:$PORT claude # lossless_guard + skill_search
+ OpenAI/opencode: OPENAI_BASE_URL=http://localhost:$PORT/v1 <client> # tool_search
+ Dashboard : headroom dashboard (http://127.0.0.1:$PORT/dashboard)
+ Raw stats : curl -s localhost:$PORT/stats | python3 -m json.tool
+
+Proxy is running (pid $PROXY_PID). Ctrl-C to stop. Logs: $LOG
+═══════════════════════════════════════════════════
+EOF
+
+wait "$PROXY_PID" || true
diff --git a/tests/test_litellm_upstream_timeout.py b/tests/test_litellm_upstream_timeout.py
new file mode 100644
index 000000000..9d6918e46
--- /dev/null
+++ b/tests/test_litellm_upstream_timeout.py
@@ -0,0 +1,70 @@
+"""Every upstream call must be bounded.
+
+There was no timeout in this backend at all. Observed 2026-08-07 under load:
+four agent workers blocked on ESTABLISHED connections for 36+ minutes while
+the proxy answered /readyz in 0.11s. No error, no retry, no log line -- the
+caller simply stops, forever, and that is indistinguishable from slow work.
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+import pytest
+
+from headroom.backends.litellm import (
+ DEFAULT_UPSTREAM_TIMEOUT,
+ UPSTREAM_TIMEOUT_ENV,
+ _upstream_timeout,
+)
+
+_SRC = Path(__file__).resolve().parents[1] / "headroom" / "backends" / "litellm.py"
+
+
+def test_every_acompletion_call_is_bounded():
+ """A new dispatch path added without a timeout reintroduces the hang.
+
+ Checked structurally rather than by mocking, because the failure mode is a
+ call site someone ADDS later -- which no mock of the existing paths sees.
+ """
+ tree = ast.parse(_SRC.read_text())
+ calls, guards = 0, 0
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.Call):
+ continue
+ fn = node.func
+ if isinstance(fn, ast.Name) and fn.id == "acompletion":
+ calls += 1
+ if (
+ isinstance(fn, ast.Attribute)
+ and fn.attr == "setdefault"
+ and node.args
+ and isinstance(node.args[0], ast.Constant)
+ and node.args[0].value == "timeout"
+ ):
+ guards += 1
+ assert calls > 0, "no acompletion call sites found -- test is stale"
+ assert guards >= calls, (
+ f"{calls} acompletion call site(s) but only {guards} timeout guard(s); "
+ "an unbounded upstream call blocks its caller forever"
+ )
+
+
+def test_a_junk_env_value_cannot_disable_the_timeout(monkeypatch):
+ """`0` means 'no timeout' to httpx, i.e. exactly the bug. So does junk."""
+ for bad in ("", "0", "-1", "nonsense", "None"):
+ monkeypatch.setenv(UPSTREAM_TIMEOUT_ENV, bad)
+ assert _upstream_timeout() == DEFAULT_UPSTREAM_TIMEOUT, bad
+
+
+def test_an_operator_can_still_tune_it(monkeypatch):
+ monkeypatch.setenv(UPSTREAM_TIMEOUT_ENV, "42.5")
+ assert _upstream_timeout() == pytest.approx(42.5)
+
+
+def test_the_default_is_generous_enough_for_real_work():
+ """Streaming: litellm expands a float across all httpx phases, so this is
+ the max gap BETWEEN CHUNKS, not a cap on total generation. A steady long
+ answer is never cut off."""
+ assert 60.0 <= DEFAULT_UPSTREAM_TIMEOUT <= 1800.0
diff --git a/tests/test_proxy/test_anthropic_recount_and_reparse_safety.py b/tests/test_proxy/test_anthropic_recount_and_reparse_safety.py
new file mode 100644
index 000000000..1e7f67fad
--- /dev/null
+++ b/tests/test_proxy/test_anthropic_recount_and_reparse_safety.py
@@ -0,0 +1,158 @@
+"""Two request-path safety nets in ``handle_anthropic_messages``.
+
+1. #2810 — the consistency re-count runs ``count_messages`` twice. Both passes
+ are CPU-bound real BPE (since #2543) and used to run directly on the event
+ loop, stalling every other in-flight request on the process (~1s on a 2.3 MB
+ body). They must run off the loop.
+2. #2768 — the byte-faithful forwarder's verification re-parse of the original
+ body is best-effort, but ``MemoryError`` is not a ``ValueError``, so on
+ 1M-context payloads it escaped and aborted an otherwise-fine request. The
+ block must never be able to fail the request.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+
+fastapi = pytest.importorskip("fastapi")
+
+from fastapi.testclient import TestClient # noqa: E402
+
+from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
+
+MESSAGES = "/v1/messages"
+MODEL = "claude-sonnet-4-6"
+
+# Only ever present in the PRE-compression snapshot, never in the outbound body.
+# Long enough to clear the handler's min-token floors.
+SENTINEL = "presnapshot-sentinel " * 500
+
+
+def _config(**overrides) -> ProxyConfig:
+ base = {
+ "optimize": True,
+ "cache_enabled": False,
+ "rate_limit_enabled": False,
+ "cost_tracking_enabled": False,
+ "mode": "token",
+ }
+ base.update(overrides)
+ return ProxyConfig(**base)
+
+
+def _upstream_200() -> MagicMock:
+ payload = {
+ "id": "msg_1",
+ "type": "message",
+ "role": "assistant",
+ "content": [{"type": "text", "text": "ok"}],
+ "model": MODEL,
+ "usage": {"input_tokens": 10, "output_tokens": 2},
+ }
+ resp = MagicMock()
+ resp.status_code = 200
+ resp.headers = {"content-type": "application/json"}
+ resp.content = json.dumps(payload).encode()
+ resp.text = json.dumps(payload)
+ resp.json.return_value = payload
+ return resp
+
+
+def test_consistency_recount_runs_off_the_event_loop(monkeypatch):
+ """No ``count_messages`` pass over the pre-compression snapshot may run on
+ the loop thread. The snapshot is identified by SENTINEL, which the pipeline
+ strips, so this pins the re-count specifically: the already-offloaded count
+ at request start also sees the sentinel and passes either way, while the
+ two re-count passes ran inline before #2810 and would fail here.
+ """
+ import headroom.tokenizers as tokenizers_mod
+
+ seen: list[bool] = [] # one entry per snapshot count: True == ran on the loop
+
+ # Patch the class, not the cached instance, so pytest restores it for us.
+ tokenizer_cls = type(tokenizers_mod.get_tokenizer(MODEL))
+ real_count = tokenizer_cls.count_messages
+
+ def counting(self, messages): # noqa: ANN001, ANN202
+ if SENTINEL in json.dumps(messages, default=str):
+ try:
+ asyncio.get_running_loop()
+ except RuntimeError:
+ seen.append(False) # worker thread — no running loop here
+ else:
+ seen.append(True) # blocking the event loop
+ return real_count(self, messages)
+
+ monkeypatch.setattr(tokenizer_cls, "count_messages", counting)
+
+ def stripping_apply(**kwargs): # noqa: ANN003, ANN202
+ """Return genuinely-changed messages with the sentinel removed."""
+ from types import SimpleNamespace
+
+ compressed = [{**m, "content": "compressed"} for m in kwargs["messages"]]
+ return SimpleNamespace(
+ messages=compressed,
+ transforms_applied=["test_strip"],
+ timing={},
+ tokens_before=100,
+ tokens_after=80,
+ waste_signals=None,
+ )
+
+ app = create_app(_config())
+ with TestClient(app) as client:
+ proxy = client.app.state.proxy
+ proxy.anthropic_pipeline.apply = MagicMock(side_effect=stripping_apply)
+ proxy._retry_request = AsyncMock(return_value=_upstream_200())
+ r = client.post(
+ MESSAGES,
+ json={
+ "model": MODEL,
+ "max_tokens": 16,
+ "messages": [{"role": "user", "content": SENTINEL}],
+ },
+ )
+
+ assert r.status_code == 200, r.text
+ assert seen, "no count_messages pass saw the snapshot; test is not exercising #2810"
+ assert not any(seen), f"{sum(seen)}/{len(seen)} snapshot counts blocked the event loop"
+
+
+def test_memoryerror_in_verification_reparse_does_not_abort_the_request(monkeypatch):
+ """A ``MemoryError`` from the best-effort original-body re-parse must be
+ swallowed (the safe fallback marks the body mutated, forcing canonical
+ re-serialization) rather than escaping and killing the request.
+ """
+ import headroom.proxy.handlers.anthropic as anthropic_mod
+
+ real_loads = json.loads
+ raised = {"n": 0}
+
+ def exploding_loads(s, *args, **kwargs): # noqa: ANN001, ANN002, ANN003, ANN202
+ # Only the verification re-parse passes the raw original body bytes.
+ if isinstance(s, (bytes, bytearray)) and b"reparse-bomb" in s:
+ raised["n"] += 1
+ raise MemoryError("simulated re-parse spike")
+ return real_loads(s, *args, **kwargs)
+
+ monkeypatch.setattr(anthropic_mod.json, "loads", exploding_loads)
+
+ app = create_app(_config(optimize=False))
+ with TestClient(app) as client:
+ proxy = client.app.state.proxy
+ proxy._retry_request = AsyncMock(return_value=_upstream_200())
+ r = client.post(
+ MESSAGES,
+ json={
+ "model": MODEL,
+ "max_tokens": 16,
+ "messages": [{"role": "user", "content": "reparse-bomb"}],
+ },
+ )
+
+ assert raised["n"] > 0, "the verification re-parse never ran; test is not exercising #2768"
+ assert r.status_code == 200, r.text