Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

docs(configuration): document cold-prefix hook flags + bound the TTL observation log (#2557)

Tejas Chopra58555c5be08d
diff --git a/docs/content/docs/cache-optimization.mdx b/docs/content/docs/cache-optimization.mdx
index baf3831d3..518be22ff 100644
--- a/docs/content/docs/cache-optimization.mdx
+++ b/docs/content/docs/cache-optimization.mdx
@@ -71,3 +71,21 @@ Keep the prefix stable in your integration layer; Headroom reports when the cach
Keep the stable prefix first, keep volatile content out of it, and treat CacheAligner warnings as a signal that the caller needs to move assembly logic.
CacheAligner surfaces prefix instability, provider caches reward byte-identical prefixes, and the caller owns the actual message layout.
+
+## Cold-prefix recompaction (when the cache lapses)
+
+Byte-identical forwarding only pays off while the prompt cache is warm. When a
+session goes idle past the provider's cache TTL, that cache is dead — so
+re-sending the prefix verbatim buys nothing, and it's the one safe moment to
+rewrite it. The **cold-prefix hook** detects a lapsed cache and recompacts the
+whole prefix (cross-turn dedupe + superseded-read drop + lossless folds) instead,
+then re-caches the smaller result. It only fires on a *confirmed*-cold turn — a
+wrong call would bust a warm cache — so it reads the real TTL (exact for Claude
+Code via its cache-control env vars; learned over time for other providers).
+
+For models that re-send **reasoning** as plain text every turn (Kimi/GLM/DeepSeek),
+a companion hook Kompresses that reasoning on warm turns and drops it on cold ones.
+
+Both are off by default. See
+[Cold-prefix hook & reasoning compaction](/docs/configuration#cold-prefix-hook--reasoning-compaction)
+for the exact flags and when each is safe to enable.
diff --git a/docs/content/docs/configuration.mdx b/docs/content/docs/configuration.mdx
index 0be97f2ab..2e301a051 100644
--- a/docs/content/docs/configuration.mdx
+++ b/docs/content/docs/configuration.mdx
@@ -315,9 +315,55 @@ headroom proxy --learn --min-evidence 3
| `HEADROOM_BETA_TRACKER_MAX_SESSIONS` | LRU capacity of the in-memory session beta tracker. Once full, the oldest session entry is evicted. | `1000` |
| `HEADROOM_MODEL_ROUTER_ENABLED` | Enable cost-aware model routing. `1`/`true`/`yes`/`on`/`enabled` turns it on and requires `HEADROOM_MODEL_ROUTES`. See [Cost-aware model routing](/docs/configuration#cost-aware-model-routing). | `off` |
| `HEADROOM_MODEL_ROUTES` | JSON array of ordered routing rules for cost-aware model routing (schema below). | -- |
+| `HEADROOM_THINKING_COMPACT` | Compact plain-text reasoning that models re-send every turn (Kimi/GLM/DeepSeek `reasoning_content` / inline `<think>`): Kompress it on warm turns, drop it on cold turns. No-op for Claude/Codex/OpenAI (encrypted reasoning). See [Cold-prefix hook](#cold-prefix-hook--reasoning-compaction). | `off` |
+| `HEADROOM_THINKING_COMPACT_KEEP_LAST` | Most-recent assistant turns whose reasoning is left intact (the active reasoning the model still uses). Only applies with `HEADROOM_THINKING_COMPACT`. | `1` |
+| `HEADROOM_COLD_RECOMPACT` | On a confirmed-cold turn (idle past the real cache TTL), recompact the whole prefix — cross-turn dedupe + superseded-read drop + lossless folds — instead of forwarding a byte-identical prefix to a dead cache. Warm turns unchanged. Pair with `HEADROOM_DEDUPE`. See [Cold-prefix hook](#cold-prefix-hook--reasoning-compaction). | `off` |
+| `HEADROOM_DEDUPE` | Whole-conversation verbatim cross-turn dedup in the router (cache-safe, information-preserving via retrieval markers). Superseded-read drop + lossless folds run without it; this adds verbatim dedup. | `off` |
+| `HEADROOM_CACHE_TTL_LEARN` | Append per-turn cache-outcome observations (provider, model, idle, hit/miss) to `cache_ttl_observations.jsonl` for the offline `headroom-cache-ttl` learner. Observation-only (no request-behavior change); respects `HEADROOM_STATELESS`; the log is size-bounded. | `off` |
+| `HEADROOM_KOMPRESS_ENDPOINT` / `HEADROOM_KOMPRESS_ENDPOINT_TOKEN` | Offload ML compression (Kompress) to a remote endpoint instead of the local ONNX model — used by reasoning compaction and the router when set. | -- |
For provider-only proxying, prefer `HEADROOM_HTTP_PROXY` over process-wide variables such as `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, or `NO_PROXY`. HTTPX reads those global variables, but Headroom also passes them through to tool executions.
+### Cold-prefix hook & reasoning compaction
+
+Two related optimizations target tokens that agent harnesses re-send every turn:
+model **reasoning** (re-billed as input on Kimi/GLM/DeepSeek and on Claude 4.6+),
+and the **prefix** itself once its prompt cache has lapsed. All flags below are
+**off by default** — the base proxy is unchanged until you opt in.
+
+**What to set for what:**
+
+| Goal | Set | Notes |
+|------|-----|-------|
+| Shrink Kimi/GLM/DeepSeek reasoning re-sent each turn | `HEADROOM_THINKING_COMPACT=1` (+ `HEADROOM_KOMPRESS_ENDPOINT` for the best ratio) | Warm turns Kompress the reasoning; cold turns drop it. No-op for Claude/Codex/OpenAI (their reasoning is an encrypted handle, already cheap on resend). |
+| Recompact a dead-cache prefix instead of re-sending it whole | `HEADROOM_COLD_RECOMPACT=1` (+ `HEADROOM_DEDUPE=1`) | Fires only when the cache has genuinely lapsed. Works in both `cache` and `token` mode. |
+| Learn each provider's real cache TTL | `HEADROOM_CACHE_TTL_LEARN=1`, then run the `headroom-cache-ttl` estimator periodically | Sharpens cold detection for providers that don't expose a TTL (Kimi/OpenAI). |
+
+**How cold detection knows the TTL.** A wrong "cold" call would recompact a
+*warm* prefix and bust the cache, so the TTL must be right:
+
+- **Claude Code:** read exactly, from the request's `cache_control.ttl` plus CC's
+ own controls — `ENABLE_PROMPT_CACHING_1H` (1h), `FORCE_PROMPT_CACHING_5M` (5m),
+ and `DISABLE_PROMPT_CACHING` (+ per-model `DISABLE_PROMPT_CACHING_<FAMILY>`),
+ which turns caching off and makes every turn a free recompaction candidate.
+- **Kimi / OpenAI / Codex:** they don't expose a TTL, so detection uses a
+ conservative default until the learner (`HEADROOM_CACHE_TTL_LEARN` + the
+ `headroom-cache-ttl` plugin) fills in the real value from observed hits/misses.
+
+**Can these be on by default?**
+
+- `HEADROOM_THINKING_COMPACT` — **stays opt-in.** It rewrites model inputs
+ (reasoning the model actively uses) and depends on Kompress, so its quality /
+ latency trade should be a deliberate choice.
+- `HEADROOM_COLD_RECOMPACT` — **opt-in today; a candidate to default for Claude
+ Code** once TTL detection is field-validated. It only fires on confirmed-cold
+ turns and recompacts losslessly, but a mis-read TTL would bust a warm cache
+ (expensive), so it waits for confidence. For Kimi/OpenAI it should stay opt-in
+ until the learner has data.
+- `HEADROOM_CACHE_TTL_LEARN` — **the safest to default on:** observation-only, a
+ size-bounded local log, and it respects `HEADROOM_STATELESS`. Kept opt-in for
+ now so nothing is written to disk unasked.
+
### Cost-aware model routing
Complementary to content compression, Headroom can rewrite the upstream model per request to stretch quota and control spend, for example by sending small, tool-free requests to a cheaper model. Routing is opt-in and disabled by default, so behavior is unchanged unless you configure it.
diff --git a/headroom/cache/ttl_observations.py b/headroom/cache/ttl_observations.py
index 148a2640f..ba3502069 100644
--- a/headroom/cache/ttl_observations.py
+++ b/headroom/cache/ttl_observations.py
@@ -32,6 +32,10 @@ _LEARN_ENV = "HEADROOM_CACHE_TTL_LEARN"
_HEADROOM_DIR = os.path.expanduser("~/.headroom")
_OBS_DEFAULT = os.path.join(_HEADROOM_DIR, "cache_ttl_observations.jsonl")
_LEARNED_DEFAULT = os.path.join(_HEADROOM_DIR, "cache_ttl_learned.json")
+# Bound the observation log so it can never grow unbounded (a prerequisite for ever
+# defaulting learning on). When exceeded, rotate to ".1" (single backup); the
+# estimator reads the current window, which is plenty for a TTL boundary.
+_OBS_MAX_BYTES = 50 * 1024 * 1024
def observations_enabled() -> bool:
@@ -57,7 +61,18 @@ def record_cache_observation(*, provider: str, model: str, attribution: Any) ->
"""
if not observations_enabled():
return
+ # Respect the global no-filesystem-writes switch (safe to default learning on
+ # without forcing writes in stateless/CI deployments).
+ if os.environ.get("HEADROOM_STATELESS", "").strip().lower() in _TRUTHY:
+ return
try:
+ path = _obs_path()
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ try:
+ if os.path.getsize(path) > _OBS_MAX_BYTES:
+ os.replace(path, path + ".1") # bounded: single-backup rotation
+ except OSError:
+ pass
row = {
"ts": round(time.time(), 3),
"provider": provider,
@@ -69,8 +84,7 @@ def record_cache_observation(*, provider: str, model: str, attribution: Any) ->
"cache_read": int(getattr(attribution, "cache_read_tokens", 0) or 0),
"expected_cached": int(getattr(attribution, "expected_cached_tokens", 0) or 0),
}
- os.makedirs(os.path.dirname(_obs_path()), exist_ok=True)
- with open(_obs_path(), "a", encoding="utf-8") as f:
+ with open(path, "a", encoding="utf-8") as f:
f.write(json.dumps(row) + "\n")
except Exception:
pass # observability must never break a request
diff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.py
index c3509bcb2..f380ab4c9 100644
--- a/headroom/proxy/handlers/openai.py
+++ b/headroom/proxy/handlers/openai.py
@@ -3915,11 +3915,16 @@ class OpenAIHandlerMixin:
# update_from_response overwrites the prior-turn state. Feeds the
# offline TTL learner (HEADROOM_CACHE_TTL_LEARN); best-effort.
try:
- if hasattr(openai_prefix_tracker, "classify_cache_miss"):
- from headroom.cache.ttl_observations import (
- record_cache_observation,
- )
+ from headroom.cache.ttl_observations import (
+ observations_enabled,
+ record_cache_observation,
+ )
+ # Only run the extra attribution when learning is enabled —
+ # otherwise this path adds nothing over the base proxy.
+ if observations_enabled() and hasattr(
+ openai_prefix_tracker, "classify_cache_miss"
+ ):
record_cache_observation(
provider="openai",
model=model,
@@ -4236,11 +4241,14 @@ class OpenAIHandlerMixin:
# Cache-TTL learning seam (see the /v1/chat path above): record the
# cache-outcome attribution before update_from_response. Best-effort.
try:
- if hasattr(openai_prefix_tracker, "classify_cache_miss"):
- from headroom.cache.ttl_observations import (
- record_cache_observation,
- )
+ from headroom.cache.ttl_observations import (
+ observations_enabled,
+ record_cache_observation,
+ )
+ if observations_enabled() and hasattr(
+ openai_prefix_tracker, "classify_cache_miss"
+ ):
record_cache_observation(
provider="openai",
model=model,