Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
perf(proxy): bound upstream calls and hot-path costs (#2852)
Tejas Chopraf624d3a00ac2
diff --git a/dashboard-cache-ttl-main.png b/dashboard-cache-ttl-main.pngnew file mode 100644index 000000000..3433b1c56Binary files /dev/null and b/dashboard-cache-ttl-main.png differdiff --git a/docs/context-mode-integration-analysis.md b/docs/context-mode-integration-analysis.mdnew file mode 100644index 000000000..fc21c7bf1--- /dev/null+++ b/docs/context-mode-integration-analysis.md@@ -0,0 +1,378 @@+# context-mode → Headroom: enterprise plugin & variant analysis++Analysis date: 2026-07-29. Sources: `/Users/tcms/demo/context-mode` @ v1.0.169, `/Users/tcms/demo/headroom` @ main.++---++## 1. Bottom line++context-mode and Headroom attack the same cost problem at **two different layers**, and they do not+overlap where it matters:++| | context-mode | Headroom |+|---|---|---|+| Interception point | agent **tool-call boundary** (host hooks + MCP) | model **API boundary** (proxy / SDK / MCP) |+| Position relative to context | **pre-context** — data never enters | **in-context** — data already entered, gets squeezed |+| Mechanism | admission control: block, redirect, sandbox, externalize | compression: crush, cache, retrieve |+| Touches the wire request | never | always |+| Loss | lossless (full content in FTS5, queryable) | lossy squeeze + hash rehydrate |++Headroom's own realignment doc identifies its correct compression target as the **live zone**:+"latest user message content + latest `tool_result` + latest `function_call_output` + latest+`local_shell_call_output`" (`REALIGNMENT/00-overview.md`, Phase B).++**That is precisely the payload context-mode intercepts one layer earlier.** Headroom Phase B is+building a Rust engine to compress the latest tool result *after* it hits the wire. context-mode+stops that tool result from being produced at all. These are complements, not competitors — and the+upstream position is strictly cheaper: nothing to compress, nothing to cache-invalidate, no+token-validation fallback needed.++Three strategic unlocks, in order of value:++1. **Cache safety.** Headroom's #1 identified bug class is prompt-cache busting from request+ mutation (5 top-tier cache-killer bugs, `REALIGNMENT/00-overview.md`). context-mode has+ *structurally zero* cache-bust risk because it never touches the request body.+2. **Subscription safety.** The realignment flags "fingerprint-class subscription-revocation+ risks" from `X-Headroom-*` header leakage, `anthropic-beta` mutation and re-serialization on+ OAuth/subscription CLIs. A hook-layer product carries none of this — it is invisible to the+ upstream. This is a *deployable-where-the-proxy-can't-go* capability.+3. **Proxy-free deployment.** Headroom's value today requires being in the API path+ (`127.0.0.1:8787`). Verified live this session: with the proxy down, `headroom_stats` returns all+ zeros and `headroom_compress` no-ops. Enterprises that cannot reroute model traffic (TLS trust,+ egress policy, subscription auth) currently get nothing. context-mode's hook+MCP model needs no+ interposition.++Zero references to context-mode exist in the Headroom tree today — clean slate.++---++## 2. context-mode: portable IP inventory++41,617 lines of TypeScript, 11 MCP tools, 18 host adapters, npm-distributed+(`[email protected]`, 8 runtime deps, esbuild-bundled).++Ranked by *how hard it would be for Headroom to rebuild*:++### Tier 1 — genuinely hard, no Headroom equivalent++**1. Cross-host hook adapter layer** — `src/adapters/**` (~10K LOC), `src/adapters/types.ts`,+`src/adapters/detect.ts` (737 lines), `configs/` (18 hosts).+Normalizes three incompatible paradigms — `json-stdio` (Claude Code, Gemini/Qwen, Copilot, Codex,+Kimi, Cursor, Kiro, Antigravity), `ts-plugin` (OpenCode, KiloCode, OpenClaw), `mcp-only` (Zed, Pi,+OMP) — behind one contract: normalized `PreToolUse` / `PostToolUse` / `PreCompact` /+`SessionStart` events, a `PlatformCapabilities` matrix, and a 5-way decision+(`allow | deny | modify | context | ask`). Per-host install, config-format, and self-heal machinery+included (`hooks/heal-partial-install.mjs`, `scripts/plugin-cache-integrity.mjs`).+*Why hard to rebuild:* the value is entirely in the accumulated per-host quirks. There is no spec to+implement against.++**2. Tool-boundary policy engine** — `src/security.ts` (889 lines).+A real policy decision point, not a regex list: glob→regex compilation, chained-command splitting+(`&&`/`;`/`|` with escape awareness), subshell extraction, deny/ask pattern ingestion from host+settings files, project-boundary containment (`evaluateProjectContainment` — Issue #852: an approved+`ctx_execute_file` cannot escape the repo via a path the user couldn't see), and a+**shell-escape scanner** (`SHELL_ESCAPE_PATTERNS`, `extractShellCommands`) that detects+`execSync`/`subprocess`/etc. embedded inside sandboxed *non-shell* code and re-evaluates the escaped+command against policy.+*Why hard to rebuild:* this is the sandbox-escape prevention layer. Getting it wrong is a CVE.++**3. Multi-language sandbox executor** — `src/executor.ts` (785), `src/runPool.ts`,+`src/exit-classify.ts`, `src/truncate.ts`.+12 languages, stdout-only egress, timeouts, background detach, output caps, exit classification.+Enforces the "Think in Code" contract: the agent programs the analysis, only the answer enters+context.++**4. Lossless externalization store** — `src/store.ts` (2,071 lines).+Dual SQLite FTS5 index — a tokenized `chunks` table *plus* a `chunks_trigram` table for+substring/identifier search where BM25 tokenization fails on code — with a `vocabulary` table and+schema migration path. Auto-externalizes any output >100 KB into FTS5 and returns a pointer.+Nothing is discarded; the model queries on demand.++### Tier 2 — valuable, but partially duplicated in Headroom++**5. Counterfactual savings accounting** — `src/session/analytics.ts` (3,085 lines),+`src/session/project-attribution.ts`, `src/session/db.ts` (1,726).+`ContextSavings`, `ThinkInCodeComparison`, `RealBytesStats`, `MultiAdapterLifetimeStats`,+`enumerateAdapterDirs()`. Measures *what would have entered context but didn't* — a different and+harder quantity than Headroom's `savings_ledger.py`, which records actual compression deltas.+Session event ledger + `tool_calls` + resume + per-project attribution.++**6. Multi-vendor pricing catalog** — `src/session/pricing.ts` + `model-prices.json`.+61 curated models × 4 rate buckets (input / output / cache-read / cache-write), refreshed from+litellm, unknown model → `null` rather than a silently wrong Claude rate.+**Overlaps `headroom/pricing/*` heavily. Do not port.**++### Tier 3 — do not port++Compression heuristics, memory/graph/relevance, telemetry transport, dashboard, install UX,+update-check. Headroom has all of these, more mature, and Phase B/H is actively consolidating them.++---++## 3. Headroom's actual extension seams++Verified entry-point groups (all `importlib.metadata`-discovered, all opt-in):++| Seam | Group | Contract | Source |+|---|---|---|---|+| Proxy extension | `headroom.proxy_extension` | `install(app: FastAPI, config: ProxyConfig) -> None` | `headroom/proxy/extensions.py:52` |+| Pipeline extension | `headroom.pipeline_extension` | `on_pipeline_event(PipelineEvent) -> PipelineEvent \| None` over 11 stages | `headroom/pipeline.py:13,68` |+| Learn plugin | `headroom.learn_plugin` | — | `headroom/learn/registry.py:44` |+| Memory text store | `headroom.memory_text` | — | `headroom/memory/config.py:41`, `factory.py:57` |+| Memory vector store | `headroom.memory_vector` | — | `headroom/memory/config.py:34` |+| Memory store | `headroom.memory_store` | — | `headroom/memory/config.py:25` |+| CCR backend | `headroom.ccr_backend` | — | `headroom/cache/compression_store.py:981` |+| Compression hooks | (subclass, not entry point) | `pre_compress` / `compute_biases` / `post_compress` | `headroom/hooks.py:1-31` |++Two things worth noting:++- `headroom/proxy/extensions.py:32` states an explicit **stability contract**: changing+ `install(app, config)` or the group name requires a deprecation cycle. This is a supported public+ seam, not an accident.+- `headroom/hooks.py:16` says outright: *"Headroom SaaS implements position-aware compression and+ cross-turn deduplication via these hooks."* The open-core split is already designed in.++**The exemplar to copy:** `plugins/headroom-oauth2/` — own `pyproject.toml`, own `LICENSE`, own+`SPEC.md`, registers on `headroom.proxy_extension`, dormant until `--proxy-extension oauth2`,+all config via env, "zero core changes." That is the enterprise plugin template.++**The precedent to copy:** `headroom/lean_ctx/installer.py` and `headroom/rtk/installer.py` —+Headroom already ships thin installers that adopt sibling products. `plugins/headroom-agent-hooks`+already installs startup hooks into Claude Code and Copilot CLI. The socket exists.++**The gap:** Headroom has *no tool-boundary interception anywhere*. It sees `tool_use`/`tool_result`+only as message content after the fact (`headroom/parser.py`, `headroom/tokenizers/*`). Its+`PipelineStage` enum has no tool-result stage. Everything context-mode does is upstream of+Headroom's earliest hook.++---++## 4. Proposed plugins & variants++Ranked by value ÷ effort.++### P1 — `headroom-recall`: FTS5+trigram lossless store as `headroom.memory_text`++**What:** port `src/store.ts` behind the existing `headroom.memory_text` seam.++**Why this first:** it is the smallest diff onto an *already-existing* contract, and it fixes a real+product limitation. Today `headroom_retrieve(hash)` requires you to *know the hash* — the tool+description literally says "hash comes from compression markers like `[N items compressed... hash=abc123]`".+With an FTS5-backed store you get `retrieve-by-query`: "what did that build log say about OOM"+instead of "paste hash abc123". The trigram index matters specifically because BM25 tokenization+loses identifiers and stack frames.++Composes rather than replaces: `compress` → return squeezed text + hash → store the *original* in+FTS5 → rehydrate by hash **or** by query. Also a natural `headroom.ccr_backend` implementation —+the realignment wants "CCR hardens: persistent backend" (Phase B), and this is one.++**Enterprise variant:** shared team store, retention/TTL policy, per-project scoping (context-mode+already has `project-attribution.ts`), audit of every retrieval.++**Effort:** medium. Reimplement in Python/Rust against Headroom's memory interface, or ship the+node store as a sidecar. Do not port the MCP tool surface — only the store.++### P2 — `headroom-admission`: tool-boundary admission control across 18 hosts++**What:** context-mode's adapter + hook layer, distributed the way `plugins/openclaw` and+`plugins/opencode` already are (TS package under `plugins/`), reporting savings into Headroom's+`savings_ledger.py` JSONL and emitting Headroom pipeline events.++**Why:** this is the strategic piece. It gives Headroom:+- a **pre-wire** enforcement point, upstream of Phase B's live-zone engine, with no cache-bust and+ no token-validation fallback required;+- coverage of **18 agent hosts** — the realignment's Phase G wants to "extend wrap CLIs (cline,+ continue, goose, openhands)"; this is that work already done, and then some;+- a deployment mode that works under **subscription auth**, where the proxy is a revocation risk.++**Enterprise value — this is the DLP story Headroom cannot currently tell.** A `curl` inside a Bash+tool call never touches the proxy, so Headroom is blind to it. context-mode blocks+`curl`/`wget`/`WebFetch`/inline `fetch()`/`requests.get` at the tool boundary and forces network+egress through `ctx_fetch_and_index`. That converts a token-savings feature into an+**egress-control** feature — a different budget line and a different buyer.++**Effort:** high, but it's mostly packaging + a reporting bridge, not a rewrite. Keep it TypeScript;+Phase H retires Python *proxy* code but explicitly preserves "CLI wrappers, RTK installer" — the+installer layer is the surviving Python, and it can shell out.++### P3 — `headroom-policy` (Enterprise, license-gated): the PDP++**What:** `src/security.ts` as a policy decision point, plus centrally-managed org rulesets.++Two attach points: the hook layer from P2 (tool-level `allow/deny/ask`), and+`headroom.pipeline_extension` at `PRE_SEND` (prompt-level policy). Feeds `headroom/audit/`.++**Enterprise features that only make sense paid:** central policy service, org-wide allow/deny+rulesets, project-boundary containment enforcement, shell-escape detection inside sandboxed code,+tamper-evident audit trail, per-team reporting. Gate it with the ELv2 license key (see §6).++**Effort:** medium. The engine exists and is tested (`tests/security/`, `src/security.ts` 889 lines);+the work is the control plane.++### P4 — `headroom-sandbox`: Think-in-Code execution++**What:** `executor.ts` exposed as a Headroom MCP tool (`headroom_execute`), 12 languages,+stdout-only.++**Why:** this is the mechanism behind context-mode's largest measured savings —+`ctx_execute_file` returns 98% savings across 315 KB of real fixtures (`BENCHMARK.md` Part 1),+versus 82% for index+search (Part 2). Programming the analysis beats compressing the output.++Must ship *with* P3: the shell-escape scanner is what stops the sandbox being an escape hatch.++**Effort:** medium-high. Runtime isolation is the hard part; `headroom` already has a `sandbox` extra+in `pyproject.toml` to build on.++### P5 — `headroom-attribution`: counterfactual savings + per-project cost++**What:** port the *methodology* from `session/analytics.ts` — `RealBytesStats`,+`ThinkInCodeComparison`, `enumerateAdapterDirs`, `project-attribution.ts` — into Headroom's+`savings_ledger` / `reporting` / `dashboard`.++**Why:** Headroom measures compression deltas (what it squeezed). context-mode measures the+counterfactual (what never entered). Enterprise buyers want the second number, sliced by team and+repo. Do **not** port `pricing.ts` — `headroom/pricing/*` already does this with litellm resolution.++**Merge, don't port.** `headroom/audit/reads.py` is already a counterfactual measurement tool over+the same Claude Code transcript corpus (see §8). It has the better mechanism taxonomy — identical+repeat, subset containment, write-readback, stale, line-number scaffolding, context residency,+cache-death windows. `analytics.ts` has the multi-host coverage and per-project attribution it+lacks. Combine the two rather than adding a third implementation.++**Effort:** low-medium, mostly a metrics-definition merge.++### Variants (packaging, not code)++- **Headroom No-Proxy Edition** — P1+P2 only, zero API interposition. Sells to buyers who cannot+ reroute model traffic and to every subscription-auth user. Removes the single biggest deployment+ blocker Headroom has.+- **Headroom Admission Control (Enterprise)** — P2+P3+P4 with a central policy plane and fleet+ enrollment across 18 hosts. Positioned as AI-agent DLP/governance, not token savings.+- **Headroom Fleet** — P5 + `enumerateAdapterDirs` for org-wide rollout state and cost reporting.++---++## 5. Evidence base++context-mode's `BENCHMARK.md`: 21 scenarios, 376 KB raw → 16.5 KB context, **96% overall**, all+fixtures captured from real tool invocations (Context7, Playwright, `gh`, vitest, tsc, nginx logs,+`git log`, analytics CSV) rather than synthetic. Honest about its weak cases — 13% on a 0.4 KB+Playwright network dump, and Part 2 openly explains why index+search only reaches 50-93% (it returns+exact code blocks rather than summaries, by design).++Test suite: 125 tests across executor/store/MCP-integration/ecosystem, plus 45 test dirs in `tests/`+covering adapters, security, session, hooks, analytics.++That's a defensible enough evidence base to reuse in Headroom's own materials, and the fixture corpus+itself is reusable for Headroom's `benchmarks/`.++---++## 6. Blockers — resolve these before writing code++**1. License incompatibility (hard blocker).**+context-mode is **Elastic License 2.0**, "Copyright 2026 Mert Koseoglu". Headroom is+**Apache-2.0**, "Copyright 2025 Headroom Contributors".++- ELv2 code **cannot** be merged into the Apache-2.0 core. Not a technicality — it would relicense+ Headroom's core.+- ELv2 forbids providing the software "to third parties as a hosted or managed service." That+ directly constrains `headroom-managed/`.+- Different copyright holders means this needs an **IP arrangement between entities**, not an+ engineering decision.++The good news: Headroom's plugin architecture is exactly the boundary that makes this tractable.+A separate package with its own `pyproject.toml` and its own `LICENSE`, registered on an entry+point — the `plugins/headroom-oauth2/` shape — can carry ELv2 while core stays Apache-2.0. ELv2 is+also the *right* license for a license-key-gated enterprise tier; it explicitly contemplates one.++Recommendation: any context-mode-derived code ships as separately-licensed plugin packages under+`plugins/`, never vendored into `headroom/`. Get the IP arrangement in writing first.++**2. Realignment collision.**+Phases A–I are ~40 PRs / 8–13 weeks and include deleting ~25K LOC. Do not open a new integration+front mid-Phase-B. P1 (`headroom.memory_text` / `ccr_backend`) is the exception — it *serves* Phase+B's "CCR hardens: persistent backend" goal rather than competing with it.++**3. Phase H direction.**+Python proxy code is being retired. Write nothing new in `headroom/proxy/`. Target the surviving+layers: installers, memory writers, CLI wrappers, and Rust.++---++## 7. Sequencing++| Order | Item | Gate |+|---|---|---|+| 0 | IP/licensing arrangement | before any code |+| 1 | P1 `headroom-recall` — FTS5 store on `memory_text`/`ccr_backend` | lands inside Phase B, serves it |+| 2 | P2 `headroom-admission` — 18-host hook layer under `plugins/` | after Phase A stabilizes |+| 3 | Variant: **No-Proxy Edition** = P1+P2 | as soon as P2 works on 3+ hosts |+| 4 | P3 `headroom-policy` (Enterprise, ELv2, key-gated) | after P2 |+| 5 | P4 `headroom-sandbox` | with P3, never before |+| 6 | P5 `headroom-attribution` | opportunistic |++---++## 8. Follow-up verification++All four items flagged as open in the first pass are now resolved.++**`headroom-managed/` is the SaaS arm, and it is unlicensed.**+`headroom-managed/pyproject.toml`: `name = "headroom-managed"`, `description = "Headroom SaaS+Platform - Managed context window optimization"`, `version = 0.1.0`. It has `app/auth.py`,+`app/middleware/`, `app/routes/`, `app/services/`, `app/models.py`, alembic migrations, and a+`pilot/`. There is **no `license` field and no LICENSE file** — i.e. proprietary by default.++This *sharpens* the §6 blocker rather than easing it. ELv2 forbids providing the software "to third+parties as a hosted or managed service." The product whose name is literally *Managed* is the one+place context-mode-derived code cannot go without an explicit commercial grant from the copyright+holder. Plan the plugin boundary so that `headroom-managed` consumes only Apache-2.0 core+interfaces, never ELv2 implementations.++**`headroom/audit/reads.py` does not overlap P3 — and it independently validates the whole thesis.**+It is a *measurement* tool, not an audit trail: it streams Claude Code `*.jsonl` transcripts to size+"the addressable bytes for each Read compression mechanism... so defaults are set from traffic, not+theory." No policy, no tamper-evidence. P3's audit trail remains a gap.++Two lines in its docstring are the most useful corroboration in either repo:++- *"context residency — how many assistant turns each Read stays in context (the multiplier on its+ prefix-cache read cost; **the case for compress-before-cache-entry**)"* — Headroom is already+ arguing, from its own traffic, for moving earlier in the pipeline. context-mode is the terminus of+ that argument: compress before **context** entry, not merely before cache entry.+- *"identical repeat — a dedup mechanism for this was prototyped and removed: it measured 0.1% of+ Read bytes on real traffic."* — Headroom has already empirically established that+ message-history-level dedup is worthless. The addressable bytes are at the tool boundary, not in+ history. That is the same conclusion the realignment reached from the cache side, arrived at+ independently from the traffic side.++It *does* overlap **P5** — `audit/reads.py` and context-mode's `session/analytics.ts` are two+independent implementations of counterfactual measurement over the same transcript corpus. Merge+them rather than porting; `audit/reads.py` has the better mechanism taxonomy, `analytics.ts` has+multi-host coverage and per-project attribution.++**No plugin-authoring docs exist.** `docs/` is a Next.js site (`app/`, `content/`, `components/`);+`wiki/` has nothing on extension authoring (only `macos-deployment.md` matched). `plugins/headroom-oauth2/SPEC.md`+remains the de-facto authoring reference — which means whichever plugin lands first sets the house+style. Worth writing the authoring doc as part of P1.++**Headroom publishes no benchmark results.** `benchmarks/` is 29 runner scripts with no committed+results artifacts, so no like-for-like number exists to compare against context-mode's 96%. The+comparison has to be run. The harness is there and is unusually strong on exactly the axis that+matters: `prefix_cache_benchmark.py`, `cache_bust_trace_report.py`, `cache_validation_bundle.py`,+`synthetic_token_cache_bust_report.py`, `proxy_mode_benchmark.py`, `agent_cost_benchmark.py`,+`real_world_agent_benchmark.py`. Use it to *prove* the §1 cache-safety claim empirically rather than+asserting it — a measured "zero cache-bust events" result is the strongest possible artifact for the+No-Proxy Edition.++**Bonus finding — the platform axes are orthogonal.**+`docs/platform-feature-matrix.json` (schema v1, updated 2026-07-06) tracks coverage across+`["linux", "macos", "windows"]` — Headroom's platform axis is **operating system**. context-mode's+platform axis is **agent host** (18 of them). Headroom tracks no host-coverage matrix at all. P2+therefore fills a dimension that does not currently exist in Headroom's own feature accounting,+which also means it needs a second matrix rather than new rows in this one.++*Process note:* six subagents were dispatched across this analysis and all six stalled at the+600-second watchdog; one reported "Bash is temporarily unavailable" before dying, so the failures+were tool-layer, not analytical. Every finding in this document was verified directly.diff --git a/headroom/backends/litellm.py b/headroom/backends/litellm.pyindex 8430789ce..60fedfc3a 100644--- a/headroom/backends/litellm.py+++ b/headroom/backends/litellm.py@@ -413,6 +413,40 @@ PROVIDER_REGISTRY: dict[str, ProviderConfig] = {}+# How long an upstream call may go silent before we give up on it.+#+# WHY THIS EXISTS. There was no timeout here at all, so a request the upstream+# never answered blocked its caller forever. Observed 2026-08-07 under load:+# four agent workers sat on ESTABLISHED connections for 36+ minutes while this+# proxy answered /readyz in 0.11s. No error, no retry, no log line -- the+# client just stops. That is the worst shape a failure can take, because it is+# indistinguishable from slow work and no supervisor can tell the difference.+#+# A float, not an httpx.Timeout, on purpose: litellm expands a float into all+# four httpx phases, so for a STREAMING call this becomes the maximum gap+# BETWEEN CHUNKS rather than a cap on total generation time. A long answer+# streaming steadily is never cut off; a stalled one dies. That is the+# semantic we want, and it falls out of the simpler type.+#+# 600s is deliberately generous -- long enough that no healthy call is at+# risk, short enough that a hang surfaces within a coffee break instead of+# never.+UPSTREAM_TIMEOUT_ENV = "HEADROOM_UPSTREAM_TIMEOUT"+DEFAULT_UPSTREAM_TIMEOUT = 600.0+++def _upstream_timeout() -> float:+ """Seconds. Never raises; a junk env value must not disable the timeout."""+ import os++ try:+ v = float(os.getenv(UPSTREAM_TIMEOUT_ENV, DEFAULT_UPSTREAM_TIMEOUT))+ except (TypeError, ValueError):+ return DEFAULT_UPSTREAM_TIMEOUT+ # 0 or negative would mean "no timeout" to httpx, which is the bug.+ return v if v > 0 else DEFAULT_UPSTREAM_TIMEOUT++def get_provider_config(provider: str) -> ProviderConfig:"""Get provider config, with fallback for unknown providers."""if provider in PROVIDER_REGISTRY:@@ -925,6 +959,9 @@ class LiteLLMBackend(Backend):logger.debug(f"LiteLLM request: model={litellm_model}")# Make the call+ # Bounded, always: an upstream that never answers must not+ # block the caller forever. setdefault so an explicit value wins.+ kwargs.setdefault("timeout", _upstream_timeout())response = await acompletion(**kwargs)# Convert to Anthropic format@@ -1055,6 +1092,9 @@ class LiteLLMBackend(Backend):kwargs["stream_options"] = {"include_usage": True}# Stream content — blocks emitted dynamically based on response+ # Bounded, always: an upstream that never answers must not+ # block the caller forever. setdefault so an explicit value wins.+ kwargs.setdefault("timeout", _upstream_timeout())response = await acompletion(**kwargs)output_tokens = 0current_block_index = -1@@ -1283,6 +1323,9 @@ class LiteLLMBackend(Backend):logger.debug(f"LiteLLM OpenAI request: model={litellm_model}")# Make the call+ # Bounded, always: an upstream that never answers must not+ # block the caller forever. setdefault so an explicit value wins.+ kwargs.setdefault("timeout", _upstream_timeout())response = await acompletion(**kwargs)# Build the usage block. LiteLLM normalizes prompt-cache stats from@@ -1452,6 +1495,9 @@ class LiteLLMBackend(Backend):elif headers.get("x-api-key"):kwargs["api_key"] = headers["x-api-key"]+ # Bounded, always: an upstream that never answers must not+ # block the caller forever. setdefault so an explicit value wins.+ kwargs.setdefault("timeout", _upstream_timeout())response = await acompletion(**kwargs)async for chunk in response:diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.pyindex 26e9f68ee..8251b3590 100644--- a/headroom/proxy/handlers/anthropic.py+++ b/headroom/proxy/handlers/anthropic.py@@ -2562,8 +2562,14 @@ class AnthropicHandlerMixin:# turn-hook fold (optimized_messages is post-hook). Runs unconditionally.try:_orig_snapshot = original_client_messages # noqa: F821 (bound at request start)- original_tokens = tokenizer.count_messages(_orig_snapshot)- optimized_tokens = tokenizer.count_messages(optimized_messages)+ # Off the event loop (#2810): both passes are CPU-bound real BPE+ # since #2543, and running them inline stalled every other+ # in-flight request on the same process (~1s on a 2.3 MB body).+ # Same tokenizer instance, so the reported values are unchanged.+ original_tokens = await asyncio.to_thread(tokenizer.count_messages, _orig_snapshot)+ optimized_tokens = await asyncio.to_thread(+ tokenizer.count_messages, optimized_messages+ )# Fold the tool-schema/desc compaction delta into BOTH endpoints so# tok_before - tok_after == tok_saved stays coherent in the PERF line# (count_messages never sees tool bytes). Same shape as the OpenAI chat@@ -2657,7 +2663,11 @@ class AnthropicHandlerMixin:parsed_original = json.loads(original_body_bytes)if parsed_original != body:body_mutation_tracker.mark_mutated("structural_diff_vs_original")- except (json.JSONDecodeError, ValueError):+ # MemoryError is not a ValueError, so a re-parse spike on 1M-context+ # bodies used to escape and abort an otherwise-fine request (#2768).+ # This block is a safety net; marking mutated is already the safe+ # outcome (it forces canonical re-serialization).+ except (json.JSONDecodeError, ValueError, MemoryError, RecursionError):body_mutation_tracker.mark_mutated("original_unparseable")if (diff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.pyindex a7d22449b..23751c65e 100644--- a/headroom/proxy/handlers/openai.py+++ b/headroom/proxy/handlers/openai.py@@ -3721,8 +3721,11 @@ class OpenAIHandlerMixin:# (result.tokens_before), which mismatches optimized_tokens (provider tokenizer)# and yields impossible tok_after>tok_before. Recount original from the# pre-compression snapshot so the message delta is on one scale.- original_tokens = tokenizer.count_messages(original_client_messages)- optimized_tokens = tokenizer.count_messages(body["messages"])+ # Off the event loop (#2810): see the matching note in the Anthropic handler.+ original_tokens = await asyncio.to_thread(+ tokenizer.count_messages, original_client_messages+ )+ optimized_tokens = await asyncio.to_thread(tokenizer.count_messages, body["messages"])if tool_tokens_before_compaction > 0:try:tool_tokens_after_compaction = tokenizer.count_text(_json_debug_dumps(tools))diff --git a/headroom/proxy/prometheus_metrics.py b/headroom/proxy/prometheus_metrics.pyindex fe477290b..889ed60be 100644--- a/headroom/proxy/prometheus_metrics.py+++ b/headroom/proxy/prometheus_metrics.py@@ -417,14 +417,14 @@ class PrometheusMetrics:return total_input_tokens, total_input_cost_usdtry:- cost_stats = self.cost_tracker.stats()+ # totals() rather than stats(): identical numbers, without the+ # 31-day cost-record walk that stats()["budget_basis"] performs and+ # this caller throws away. See CostTracker.totals.+ tracked_input_tokens, tracked_input_cost_usd = self.cost_tracker.totals()except Exception:logger.debug("Failed to read cost tracker totals for savings history", exc_info=True)return total_input_tokens, total_input_cost_usd- tracked_input_tokens = cost_stats.get("total_input_tokens")- tracked_input_cost_usd = cost_stats.get("total_input_cost_usd")-if tracked_input_tokens is not None:try:total_input_tokens = self._savings_tracker_input_tokens_offset + max(diff --git a/run-all-plugins.sh b/run-all-plugins.shnew file mode 100755index 000000000..f8b9b2ec9--- /dev/null+++ b/run-all-plugins.sh@@ -0,0 +1,171 @@+#!/usr/bin/env bash+# ─────────────────────────────────────────────────────────────────────────────+# run-all-plugins.sh — install + configure + run the Headroom proxy with ALL 5+# enterprise plugins, the coding savings-profile, and ML compression offloaded+# to the Kompress-v2 Modal endpoint. Then confirm everything loaded.+#+# Plugins : lossless_guard, skill_search, observability, tier_router, tool_search+# Extra : headroom-ai[sandbox] (torch-free proxy; ML offloaded to Modal)+# Profile : coding (HEADROOM_SAVINGS_PROFILE) + cache mode (prefix-cache safe)+#+# Secrets are SOURCED from ~/env.txt and ~/.headroom/plugins.env — never inlined.+# Re-runnable: install is skipped when already satisfied (FORCE_INSTALL=1 forces).+# ─────────────────────────────────────────────────────────────────────────────+set -euo pipefail++HR=/Users/tcms/demo/headroom+VENV="$HR/.venv"+PORT="${HEADROOM_PORT:-8787}"+ENV_TXT="${ENV_TXT:-$HOME/env.txt}"+PLUGINS_ENV="$HOME/.headroom/plugins.env"+LOG="${HEADROOM_LOG:-$HOME/.headroom/logs/proxy-all-plugins.log}"+mkdir -p "$(dirname "$LOG")"++# ── 1. venv ──────────────────────────────────────────────────────────────────+# `python`/`pip`/`uv` are broken system-wide on this box — always use the venv,+# and `python -m pip` (the .venv/bin/pip shim is broken too).+# shellcheck disable=SC1091+source "$VENV/bin/activate"+PY="$VENV/bin/python"++# ── 2. install (guarded) ──────────────────────────────────────────────────────+# headroom-ai[sandbox] pulls proxy,code,relevance,reports,otel,html,mcp,spreadsheet+# (all torch-free — heavy ML is offloaded to the Modal Kompress endpoint below).+# The 5 plugins install --no-deps so pip won't drag PyPI's headroom-ai over the+# local editable one; headroom-license is their shared Ed25519 verifier.+need_install=1+if [ "${FORCE_INSTALL:-0}" != "1" ]; then+ n=$("$PY" -c 'import opentelemetry; from headroom.proxy.extensions import discover; print(len(list(discover())))' 2>/dev/null || echo 0)+ [ "$n" = "5" ] && need_install=0+fi+if [ "$need_install" = "1" ]; then+ avail=$(df -g "$HR" 2>/dev/null | awk 'NR==2{print $4}')+ echo "▶ disk: ${avail:-?}Gi free before install"+ if [ -n "$avail" ] && [ "$avail" -lt 2 ]; then+ echo "!! <2Gi free — aborting before heavy install (free space, then re-run)"; exit 1+ fi+ echo "▶ installing pip+maturin, then headroom-ai[sandbox] + license + 5 plugins (editable)…"+ "$PY" -m pip install -U pip maturin+ # litellm >=1.92 ships an sdist-only Rust bridge whose AWS-SDK crates need rustc>=1.94.1;+ # the default rustup toolchain here is older (pip builds litellm in a temp dir that misses+ # the repo's 1.95 pin), so pin to the last pure-Python wheel line (1.91.4). Satisfies+ # headroom's litellm>=1.86.2,<2.0 and skips the Rust build entirely.+ "$PY" -m pip install "litellm<1.92"+ "$PY" -m pip install -e "${HR}[sandbox]" "litellm<1.92"+ "$PY" -m pip install -e /Users/tcms/demo/headroom-license+ for p in lossless-guard skill-search observability tier-router tool-search; do+ "$PY" -m pip install -e "/Users/tcms/demo/headroom-${p}" --no-deps+ done+else+ echo "▶ install satisfied (5 extensions discovered) — skipping (FORCE_INSTALL=1 to force)"+fi++# ── 3. secrets from ~/env.txt ─────────────────────────────────────────────────+# Provides: OPENAI_API_KEY, ANTHROPIC_API_KEY, FIREWORKS_API_KEY (upstream creds);+# LANGFUSE_{PUBLIC,SECRET}_KEY + LANGFUSE_BASE_URL (observability sink);+# HEADROOM_KOMPRESS_ENDPOINT + _TOKEN (Modal ML offload).+[ -f "$ENV_TXT" ] || { echo "!! $ENV_TXT not found"; exit 1; }+set -a; # shellcheck disable=SC1090+source "$ENV_TXT"; set +a++# ── 4. plugin license (Ed25519, offline, wildcard) ────────────────────────────+# HEADROOM_LICENSE + HEADROOM_LICENSE_PUBKEY. This is SEPARATE from the OSS cloud+# key (HEADROOM_LICENSE_KEY) — the banner will still say "OSS (no license key)",+# but each plugin prints "license accepted". Fallback: skip verification entirely.+if [ -f "$PLUGINS_ENV" ]; then+ set -a; # shellcheck disable=SC1090+ source "$PLUGINS_ENV"; set +a+else+ echo "▶ $PLUGINS_ENV missing — using dev license bypass"+ export HEADROOM_LICENSE_DEV=1+fi++# ── 5. Kompress ML offload → Modal ────────────────────────────────────────────+# Setting HEADROOM_KOMPRESS_ENDPOINT (+_TOKEN) alone routes Kompress inference to+# the Modal endpoint (content_router._get_kompress_remote). No other flag needed;+# HEADROOM_COMPRESS_ALLOW_REMOTE is a different thing (remote upstreams, not this).+: "${HEADROOM_KOMPRESS_ENDPOINT:?must be set in $ENV_TXT}"+export HEADROOM_KOMPRESS_ENDPOINT_TOKEN="${HEADROOM_KOMPRESS_ENDPOINT_TOKEN:-}"++# ── 6. observability sink → Langfuse + spend attribution ──────────────────────+# HEADROOM_LANGFUSE_ENABLED must be explicitly truthy (LANGFUSE_* creds come from+# env.txt). Traces (agent.turn / llm.turn spans with gen_ai.usage.cost) land in+# Langfuse. Spend is opt-in: HEADROOM_MODEL_PRICES is {model-substr:{in,out}} in+# USD per 1K tokens. (Per-request identity — org/team/user/session — is supplied+# by the CLIENT via x-headroom-* headers, not settable here.)+export HEADROOM_LANGFUSE_ENABLED=1+export HEADROOM_LANGFUSE_SERVICE_NAME=headroom-proxy+export HEADROOM_MODEL_PRICES='{"claude-opus":{"in":0.015,"out":0.075},"claude-sonnet":{"in":0.003,"out":0.015},"gpt-5":{"in":0.00125,"out":0.01},"gpt-4":{"in":0.003,"out":0.012}}'+# Metrics (counters) need a separate OTLP endpoint — none in env.txt, so left off:+# export HEADROOM_OTEL_METRICS_ENABLED=1 HEADROOM_OTEL_METRICS_ENDPOINT=http://localhost:4318++# ── 7. tier_router ─────────────────────────────────────────────────────────────+# Only stamps service_tier on the wire (no token delta). OpenAI 'flex' is only+# auto-selected for models declared eligible here. Anthropic tiers are a no-op by+# default. Clients force a tier with x-headroom-tier / x-headroom-background: 1.+export HEADROOM_TIER_FLEX_MODELS="${HEADROOM_TIER_FLEX_MODELS:-gpt-5,gpt-4.1,o4-mini}"++# ── 8. plugin tuning (defaults shown; override as needed) ─────────────────────+# skill_search fires on Anthropic w/ >=min skills; tool_search on synthetic-tier+# providers w/ >=min tools; lossless_guard lossy tier is opt-in (kept OFF).+export HEADROOM_SKILL_SEARCH_MIN_SKILLS="${HEADROOM_SKILL_SEARCH_MIN_SKILLS:-8}"+export HEADROOM_TOOL_SEARCH_MIN_TOOLS="${HEADROOM_TOOL_SEARCH_MIN_TOOLS:-5}"+# export HEADROOM_LOSSLESS_GUARD_LOSSY=1 # opt-in irreversible Bash-noise drop++# ── 9. coding profile + mode ───────────────────────────────────────────────────+# savings_profile=coding tunes the pipeline for coding-agent traffic; cache mode+# freezes prior turns to preserve the provider prefix-cache (what coding wants).+export HEADROOM_SAVINGS_PROFILE=coding++# ── 10. run + confirm ──────────────────────────────────────────────────────────+cleanup() { [ -n "${PROXY_PID:-}" ] && kill "$PROXY_PID" 2>/dev/null || true; }+trap cleanup INT TERM EXIT++echo "▶ starting proxy on :$PORT (profile=coding, mode=cache, all 5 extensions)…"+headroom proxy --port "$PORT" --mode cache --proxy-extension '*' > "$LOG" 2>&1 &+PROXY_PID=$!++# wait for readiness (no foreground sleep on this harness)+curl -s --retry 40 --retry-delay 1 --retry-all-errors --max-time 60 \+ "http://127.0.0.1:$PORT/health" >/dev/null 2>&1 || true++echo+echo "══════════════════ CONFIRMATION ══════════════════"+echo "── extensions loaded (from $LOG) ──"+grep -iE "Extensions:|license accepted|installed \(" "$LOG" | sed 's/^/ /' || true++echo "── Modal Kompress endpoint reachable? ──"+code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "$HEADROOM_KOMPRESS_ENDPOINT" || echo "unreachable")+echo " $HEADROOM_KOMPRESS_ENDPOINT -> HTTP $code (any response = up; offload runs on real traffic)"++echo "── /stats surfaces (empty until traffic flows) ──"+curl -s --max-time 5 "http://127.0.0.1:$PORT/stats" | "$PY" -c '+import sys,json+d=json.load(sys.stdin)+print(" extension_savings :", d.get("extension_savings"))+print(" by_layer :", list(d.get("savings",{}).get("by_layer",{})))+print(" tokens_saved_by_strat:", d.get("tokens_saved_by_strategy"))+print(" otel.enabled :", d.get("otel",{}).get("enabled"))+print(" langfuse.enabled :", d.get("langfuse",{}).get("enabled"))+' 2>/dev/null || echo " (stats not ready)"++cat <<EOF++── where each effect shows up ──+ lossless_guard -> dashboard (compression layer) + /stats.tokens_saved_by_strategy+ skill_search -> /stats.extension_savings (NOT dashboard) — Anthropic client, >=8 skills+ tool_search -> /stats.extension_savings (NOT dashboard) — synthetic-tier client, >=5 tools+ observability -> Langfuse UI (spans + gen_ai.usage.cost) — send x-headroom-org/user/session+ tier_router -> service_tier on the wire / provider bill (no token delta)++── drive traffic (two clients — they exercise different plugins) ──+ Claude Code : ANTHROPIC_BASE_URL=http://localhost:$PORT claude # lossless_guard + skill_search+ OpenAI/opencode: OPENAI_BASE_URL=http://localhost:$PORT/v1 <client> # tool_search+ Dashboard : headroom dashboard (http://127.0.0.1:$PORT/dashboard)+ Raw stats : curl -s localhost:$PORT/stats | python3 -m json.tool++Proxy is running (pid $PROXY_PID). Ctrl-C to stop. Logs: $LOG+═══════════════════════════════════════════════════+EOF++wait "$PROXY_PID" || truediff --git a/tests/test_litellm_upstream_timeout.py b/tests/test_litellm_upstream_timeout.pynew file mode 100644index 000000000..9d6918e46--- /dev/null+++ b/tests/test_litellm_upstream_timeout.py@@ -0,0 +1,70 @@+"""Every upstream call must be bounded.++There was no timeout in this backend at all. Observed 2026-08-07 under load:+four agent workers blocked on ESTABLISHED connections for 36+ minutes while+the proxy answered /readyz in 0.11s. No error, no retry, no log line -- the+caller simply stops, forever, and that is indistinguishable from slow work.+"""++from __future__ import annotations++import ast+from pathlib import Path++import pytest++from headroom.backends.litellm import (+ DEFAULT_UPSTREAM_TIMEOUT,+ UPSTREAM_TIMEOUT_ENV,+ _upstream_timeout,+)++_SRC = Path(__file__).resolve().parents[1] / "headroom" / "backends" / "litellm.py"+++def test_every_acompletion_call_is_bounded():+ """A new dispatch path added without a timeout reintroduces the hang.++ Checked structurally rather than by mocking, because the failure mode is a+ call site someone ADDS later -- which no mock of the existing paths sees.+ """+ tree = ast.parse(_SRC.read_text())+ calls, guards = 0, 0+ for node in ast.walk(tree):+ if not isinstance(node, ast.Call):+ continue+ fn = node.func+ if isinstance(fn, ast.Name) and fn.id == "acompletion":+ calls += 1+ if (+ isinstance(fn, ast.Attribute)+ and fn.attr == "setdefault"+ and node.args+ and isinstance(node.args[0], ast.Constant)+ and node.args[0].value == "timeout"+ ):+ guards += 1+ assert calls > 0, "no acompletion call sites found -- test is stale"+ assert guards >= calls, (+ f"{calls} acompletion call site(s) but only {guards} timeout guard(s); "+ "an unbounded upstream call blocks its caller forever"+ )+++def test_a_junk_env_value_cannot_disable_the_timeout(monkeypatch):+ """`0` means 'no timeout' to httpx, i.e. exactly the bug. So does junk."""+ for bad in ("", "0", "-1", "nonsense", "None"):+ monkeypatch.setenv(UPSTREAM_TIMEOUT_ENV, bad)+ assert _upstream_timeout() == DEFAULT_UPSTREAM_TIMEOUT, bad+++def test_an_operator_can_still_tune_it(monkeypatch):+ monkeypatch.setenv(UPSTREAM_TIMEOUT_ENV, "42.5")+ assert _upstream_timeout() == pytest.approx(42.5)+++def test_the_default_is_generous_enough_for_real_work():+ """Streaming: litellm expands a float across all httpx phases, so this is+ the max gap BETWEEN CHUNKS, not a cap on total generation. A steady long+ answer is never cut off."""+ assert 60.0 <= DEFAULT_UPSTREAM_TIMEOUT <= 1800.0diff --git a/tests/test_proxy/test_anthropic_recount_and_reparse_safety.py b/tests/test_proxy/test_anthropic_recount_and_reparse_safety.pynew file mode 100644index 000000000..1e7f67fad--- /dev/null+++ b/tests/test_proxy/test_anthropic_recount_and_reparse_safety.py@@ -0,0 +1,158 @@+"""Two request-path safety nets in ``handle_anthropic_messages``.++1. #2810 — the consistency re-count runs ``count_messages`` twice. Both passes+ are CPU-bound real BPE (since #2543) and used to run directly on the event+ loop, stalling every other in-flight request on the process (~1s on a 2.3 MB+ body). They must run off the loop.+2. #2768 — the byte-faithful forwarder's verification re-parse of the original+ body is best-effort, but ``MemoryError`` is not a ``ValueError``, so on+ 1M-context payloads it escaped and aborted an otherwise-fine request. The+ block must never be able to fail the request.+"""++from __future__ import annotations++import asyncio+import json+from unittest.mock import AsyncMock, MagicMock++import pytest++fastapi = pytest.importorskip("fastapi")++from fastapi.testclient import TestClient # noqa: E402++from headroom.proxy.server import ProxyConfig, create_app # noqa: E402++MESSAGES = "/v1/messages"+MODEL = "claude-sonnet-4-6"++# Only ever present in the PRE-compression snapshot, never in the outbound body.+# Long enough to clear the handler's min-token floors.+SENTINEL = "presnapshot-sentinel " * 500+++def _config(**overrides) -> ProxyConfig:+ base = {+ "optimize": True,+ "cache_enabled": False,+ "rate_limit_enabled": False,+ "cost_tracking_enabled": False,+ "mode": "token",+ }+ base.update(overrides)+ return ProxyConfig(**base)+++def _upstream_200() -> MagicMock:+ payload = {+ "id": "msg_1",+ "type": "message",+ "role": "assistant",+ "content": [{"type": "text", "text": "ok"}],+ "model": MODEL,+ "usage": {"input_tokens": 10, "output_tokens": 2},+ }+ resp = MagicMock()+ resp.status_code = 200+ resp.headers = {"content-type": "application/json"}+ resp.content = json.dumps(payload).encode()+ resp.text = json.dumps(payload)+ resp.json.return_value = payload+ return resp+++def test_consistency_recount_runs_off_the_event_loop(monkeypatch):+ """No ``count_messages`` pass over the pre-compression snapshot may run on+ the loop thread. The snapshot is identified by SENTINEL, which the pipeline+ strips, so this pins the re-count specifically: the already-offloaded count+ at request start also sees the sentinel and passes either way, while the+ two re-count passes ran inline before #2810 and would fail here.+ """+ import headroom.tokenizers as tokenizers_mod++ seen: list[bool] = [] # one entry per snapshot count: True == ran on the loop++ # Patch the class, not the cached instance, so pytest restores it for us.+ tokenizer_cls = type(tokenizers_mod.get_tokenizer(MODEL))+ real_count = tokenizer_cls.count_messages++ def counting(self, messages): # noqa: ANN001, ANN202+ if SENTINEL in json.dumps(messages, default=str):+ try:+ asyncio.get_running_loop()+ except RuntimeError:+ seen.append(False) # worker thread — no running loop here+ else:+ seen.append(True) # blocking the event loop+ return real_count(self, messages)++ monkeypatch.setattr(tokenizer_cls, "count_messages", counting)++ def stripping_apply(**kwargs): # noqa: ANN003, ANN202+ """Return genuinely-changed messages with the sentinel removed."""+ from types import SimpleNamespace++ compressed = [{**m, "content": "compressed"} for m in kwargs["messages"]]+ return SimpleNamespace(+ messages=compressed,+ transforms_applied=["test_strip"],+ timing={},+ tokens_before=100,+ tokens_after=80,+ waste_signals=None,+ )++ app = create_app(_config())+ with TestClient(app) as client:+ proxy = client.app.state.proxy+ proxy.anthropic_pipeline.apply = MagicMock(side_effect=stripping_apply)+ proxy._retry_request = AsyncMock(return_value=_upstream_200())+ r = client.post(+ MESSAGES,+ json={+ "model": MODEL,+ "max_tokens": 16,+ "messages": [{"role": "user", "content": SENTINEL}],+ },+ )++ assert r.status_code == 200, r.text+ assert seen, "no count_messages pass saw the snapshot; test is not exercising #2810"+ assert not any(seen), f"{sum(seen)}/{len(seen)} snapshot counts blocked the event loop"+++def test_memoryerror_in_verification_reparse_does_not_abort_the_request(monkeypatch):+ """A ``MemoryError`` from the best-effort original-body re-parse must be+ swallowed (the safe fallback marks the body mutated, forcing canonical+ re-serialization) rather than escaping and killing the request.+ """+ import headroom.proxy.handlers.anthropic as anthropic_mod++ real_loads = json.loads+ raised = {"n": 0}++ def exploding_loads(s, *args, **kwargs): # noqa: ANN001, ANN002, ANN003, ANN202+ # Only the verification re-parse passes the raw original body bytes.+ if isinstance(s, (bytes, bytearray)) and b"reparse-bomb" in s:+ raised["n"] += 1+ raise MemoryError("simulated re-parse spike")+ return real_loads(s, *args, **kwargs)++ monkeypatch.setattr(anthropic_mod.json, "loads", exploding_loads)++ app = create_app(_config(optimize=False))+ with TestClient(app) as client:+ proxy = client.app.state.proxy+ proxy._retry_request = AsyncMock(return_value=_upstream_200())+ r = client.post(+ MESSAGES,+ json={+ "model": MODEL,+ "max_tokens": 16,+ "messages": [{"role": "user", "content": "reparse-bomb"}],+ },+ )++ assert raised["n"] > 0, "the verification re-parse never ran; test is not exercising #2768"+ assert r.status_code == 200, r.text