Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
docs(configuration): document cold-prefix hook flags + bound the TTL observation log (#2557)
Tejas Chopra58555c5be08d
diff --git a/docs/content/docs/cache-optimization.mdx b/docs/content/docs/cache-optimization.mdxindex baf3831d3..518be22ff 100644--- a/docs/content/docs/cache-optimization.mdx+++ b/docs/content/docs/cache-optimization.mdx@@ -71,3 +71,21 @@ Keep the prefix stable in your integration layer; Headroom reports when the cachKeep the stable prefix first, keep volatile content out of it, and treat CacheAligner warnings as a signal that the caller needs to move assembly logic.CacheAligner surfaces prefix instability, provider caches reward byte-identical prefixes, and the caller owns the actual message layout.++## Cold-prefix recompaction (when the cache lapses)++Byte-identical forwarding only pays off while the prompt cache is warm. When a+session goes idle past the provider's cache TTL, that cache is dead — so+re-sending the prefix verbatim buys nothing, and it's the one safe moment to+rewrite it. The **cold-prefix hook** detects a lapsed cache and recompacts the+whole prefix (cross-turn dedupe + superseded-read drop + lossless folds) instead,+then re-caches the smaller result. It only fires on a *confirmed*-cold turn — a+wrong call would bust a warm cache — so it reads the real TTL (exact for Claude+Code via its cache-control env vars; learned over time for other providers).++For models that re-send **reasoning** as plain text every turn (Kimi/GLM/DeepSeek),+a companion hook Kompresses that reasoning on warm turns and drops it on cold ones.++Both are off by default. See+[Cold-prefix hook & reasoning compaction](/docs/configuration#cold-prefix-hook--reasoning-compaction)+for the exact flags and when each is safe to enable.diff --git a/docs/content/docs/configuration.mdx b/docs/content/docs/configuration.mdxindex 0be97f2ab..2e301a051 100644--- a/docs/content/docs/configuration.mdx+++ b/docs/content/docs/configuration.mdx@@ -315,9 +315,55 @@ headroom proxy --learn --min-evidence 3| `HEADROOM_BETA_TRACKER_MAX_SESSIONS` | LRU capacity of the in-memory session beta tracker. Once full, the oldest session entry is evicted. | `1000` || `HEADROOM_MODEL_ROUTER_ENABLED` | Enable cost-aware model routing. `1`/`true`/`yes`/`on`/`enabled` turns it on and requires `HEADROOM_MODEL_ROUTES`. See [Cost-aware model routing](/docs/configuration#cost-aware-model-routing). | `off` || `HEADROOM_MODEL_ROUTES` | JSON array of ordered routing rules for cost-aware model routing (schema below). | -- |+| `HEADROOM_THINKING_COMPACT` | Compact plain-text reasoning that models re-send every turn (Kimi/GLM/DeepSeek `reasoning_content` / inline `<think>`): Kompress it on warm turns, drop it on cold turns. No-op for Claude/Codex/OpenAI (encrypted reasoning). See [Cold-prefix hook](#cold-prefix-hook--reasoning-compaction). | `off` |+| `HEADROOM_THINKING_COMPACT_KEEP_LAST` | Most-recent assistant turns whose reasoning is left intact (the active reasoning the model still uses). Only applies with `HEADROOM_THINKING_COMPACT`. | `1` |+| `HEADROOM_COLD_RECOMPACT` | On a confirmed-cold turn (idle past the real cache TTL), recompact the whole prefix — cross-turn dedupe + superseded-read drop + lossless folds — instead of forwarding a byte-identical prefix to a dead cache. Warm turns unchanged. Pair with `HEADROOM_DEDUPE`. See [Cold-prefix hook](#cold-prefix-hook--reasoning-compaction). | `off` |+| `HEADROOM_DEDUPE` | Whole-conversation verbatim cross-turn dedup in the router (cache-safe, information-preserving via retrieval markers). Superseded-read drop + lossless folds run without it; this adds verbatim dedup. | `off` |+| `HEADROOM_CACHE_TTL_LEARN` | Append per-turn cache-outcome observations (provider, model, idle, hit/miss) to `cache_ttl_observations.jsonl` for the offline `headroom-cache-ttl` learner. Observation-only (no request-behavior change); respects `HEADROOM_STATELESS`; the log is size-bounded. | `off` |+| `HEADROOM_KOMPRESS_ENDPOINT` / `HEADROOM_KOMPRESS_ENDPOINT_TOKEN` | Offload ML compression (Kompress) to a remote endpoint instead of the local ONNX model — used by reasoning compaction and the router when set. | -- |For provider-only proxying, prefer `HEADROOM_HTTP_PROXY` over process-wide variables such as `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, or `NO_PROXY`. HTTPX reads those global variables, but Headroom also passes them through to tool executions.+### Cold-prefix hook & reasoning compaction++Two related optimizations target tokens that agent harnesses re-send every turn:+model **reasoning** (re-billed as input on Kimi/GLM/DeepSeek and on Claude 4.6+),+and the **prefix** itself once its prompt cache has lapsed. All flags below are+**off by default** — the base proxy is unchanged until you opt in.++**What to set for what:**++| Goal | Set | Notes |+|------|-----|-------|+| Shrink Kimi/GLM/DeepSeek reasoning re-sent each turn | `HEADROOM_THINKING_COMPACT=1` (+ `HEADROOM_KOMPRESS_ENDPOINT` for the best ratio) | Warm turns Kompress the reasoning; cold turns drop it. No-op for Claude/Codex/OpenAI (their reasoning is an encrypted handle, already cheap on resend). |+| Recompact a dead-cache prefix instead of re-sending it whole | `HEADROOM_COLD_RECOMPACT=1` (+ `HEADROOM_DEDUPE=1`) | Fires only when the cache has genuinely lapsed. Works in both `cache` and `token` mode. |+| Learn each provider's real cache TTL | `HEADROOM_CACHE_TTL_LEARN=1`, then run the `headroom-cache-ttl` estimator periodically | Sharpens cold detection for providers that don't expose a TTL (Kimi/OpenAI). |++**How cold detection knows the TTL.** A wrong "cold" call would recompact a+*warm* prefix and bust the cache, so the TTL must be right:++- **Claude Code:** read exactly, from the request's `cache_control.ttl` plus CC's+ own controls — `ENABLE_PROMPT_CACHING_1H` (1h), `FORCE_PROMPT_CACHING_5M` (5m),+ and `DISABLE_PROMPT_CACHING` (+ per-model `DISABLE_PROMPT_CACHING_<FAMILY>`),+ which turns caching off and makes every turn a free recompaction candidate.+- **Kimi / OpenAI / Codex:** they don't expose a TTL, so detection uses a+ conservative default until the learner (`HEADROOM_CACHE_TTL_LEARN` + the+ `headroom-cache-ttl` plugin) fills in the real value from observed hits/misses.++**Can these be on by default?**++- `HEADROOM_THINKING_COMPACT` — **stays opt-in.** It rewrites model inputs+ (reasoning the model actively uses) and depends on Kompress, so its quality /+ latency trade should be a deliberate choice.+- `HEADROOM_COLD_RECOMPACT` — **opt-in today; a candidate to default for Claude+ Code** once TTL detection is field-validated. It only fires on confirmed-cold+ turns and recompacts losslessly, but a mis-read TTL would bust a warm cache+ (expensive), so it waits for confidence. For Kimi/OpenAI it should stay opt-in+ until the learner has data.+- `HEADROOM_CACHE_TTL_LEARN` — **the safest to default on:** observation-only, a+ size-bounded local log, and it respects `HEADROOM_STATELESS`. Kept opt-in for+ now so nothing is written to disk unasked.+### Cost-aware model routingComplementary to content compression, Headroom can rewrite the upstream model per request to stretch quota and control spend, for example by sending small, tool-free requests to a cheaper model. Routing is opt-in and disabled by default, so behavior is unchanged unless you configure it.diff --git a/headroom/cache/ttl_observations.py b/headroom/cache/ttl_observations.pyindex 148a2640f..ba3502069 100644--- a/headroom/cache/ttl_observations.py+++ b/headroom/cache/ttl_observations.py@@ -32,6 +32,10 @@ _LEARN_ENV = "HEADROOM_CACHE_TTL_LEARN"_HEADROOM_DIR = os.path.expanduser("~/.headroom")_OBS_DEFAULT = os.path.join(_HEADROOM_DIR, "cache_ttl_observations.jsonl")_LEARNED_DEFAULT = os.path.join(_HEADROOM_DIR, "cache_ttl_learned.json")+# Bound the observation log so it can never grow unbounded (a prerequisite for ever+# defaulting learning on). When exceeded, rotate to ".1" (single backup); the+# estimator reads the current window, which is plenty for a TTL boundary.+_OBS_MAX_BYTES = 50 * 1024 * 1024def observations_enabled() -> bool:@@ -57,7 +61,18 @@ def record_cache_observation(*, provider: str, model: str, attribution: Any) ->"""if not observations_enabled():return+ # Respect the global no-filesystem-writes switch (safe to default learning on+ # without forcing writes in stateless/CI deployments).+ if os.environ.get("HEADROOM_STATELESS", "").strip().lower() in _TRUTHY:+ returntry:+ path = _obs_path()+ os.makedirs(os.path.dirname(path), exist_ok=True)+ try:+ if os.path.getsize(path) > _OBS_MAX_BYTES:+ os.replace(path, path + ".1") # bounded: single-backup rotation+ except OSError:+ passrow = {"ts": round(time.time(), 3),"provider": provider,@@ -69,8 +84,7 @@ def record_cache_observation(*, provider: str, model: str, attribution: Any) ->"cache_read": int(getattr(attribution, "cache_read_tokens", 0) or 0),"expected_cached": int(getattr(attribution, "expected_cached_tokens", 0) or 0),}- os.makedirs(os.path.dirname(_obs_path()), exist_ok=True)- with open(_obs_path(), "a", encoding="utf-8") as f:+ with open(path, "a", encoding="utf-8") as f:f.write(json.dumps(row) + "\n")except Exception:pass # observability must never break a requestdiff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.pyindex c3509bcb2..f380ab4c9 100644--- a/headroom/proxy/handlers/openai.py+++ b/headroom/proxy/handlers/openai.py@@ -3915,11 +3915,16 @@ class OpenAIHandlerMixin:# update_from_response overwrites the prior-turn state. Feeds the# offline TTL learner (HEADROOM_CACHE_TTL_LEARN); best-effort.try:- if hasattr(openai_prefix_tracker, "classify_cache_miss"):- from headroom.cache.ttl_observations import (- record_cache_observation,- )+ from headroom.cache.ttl_observations import (+ observations_enabled,+ record_cache_observation,+ )+ # Only run the extra attribution when learning is enabled —+ # otherwise this path adds nothing over the base proxy.+ if observations_enabled() and hasattr(+ openai_prefix_tracker, "classify_cache_miss"+ ):record_cache_observation(provider="openai",model=model,@@ -4236,11 +4241,14 @@ class OpenAIHandlerMixin:# Cache-TTL learning seam (see the /v1/chat path above): record the# cache-outcome attribution before update_from_response. Best-effort.try:- if hasattr(openai_prefix_tracker, "classify_cache_miss"):- from headroom.cache.ttl_observations import (- record_cache_observation,- )+ from headroom.cache.ttl_observations import (+ observations_enabled,+ record_cache_observation,+ )+ if observations_enabled() and hasattr(+ openai_prefix_tracker, "classify_cache_miss"+ ):record_cache_observation(provider="openai",model=model,