Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(proxy): make CCR multi-worker warning conditional on backend (#770)

Patrick Ad76a7296df12
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7545d2140..3414bdb77 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -78,6 +78,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Bug Fixes
* **codex:** respect `CODEX_HOME` when `headroom wrap codex` writes provider, MCP, memory, backup, and global `AGENTS.md` config, and warn when `unwrap codex` may be looking at the default Codex home because `CODEX_HOME` is unset.
+* **proxy:** multi-worker CCR warning is now conditional on backend — when `HEADROOM_CCR_BACKEND` is unset (default `InMemoryBackend`, per-process), the startup warning includes CCR retrieval failures and suggests `HEADROOM_CCR_BACKEND=sqlite`; when a cross-worker backend is already configured, the warning covers only the remaining per-worker stores (compression cache, prefix tracker, TOIN, CostTracker). Updated `RUST_DEV.md` to accurately document Python `CompressionStore` as per-process by default.
* **deps:** move `gunicorn` to `[proxy-prod]` extra with `sys_platform != 'win32'` guard; removed from `[proxy]` to avoid forcing a Unix-only package on dev, CI, and Windows users ([#537](https://github.com/chopratejas/headroom/pull/537))
* **startup:** suppress proxy startup log noise -- litellm banner, trafilatura parse errors, HuggingFace Hub unauthenticated warnings, tiktoken fallback warning, and httpx INFO lines from sentence_transformers HEAD checks. Affected files: `headroom/providers/litellm.py`, `headroom/transforms/html_extractor.py`, `headroom/memory/adapters/embedders.py`, `headroom/providers/anthropic.py`, `headroom/providers/registry.py`, `headroom/image/onnx_router.py`, `headroom/transforms/kompress_compressor.py`.
diff --git a/RUST_DEV.md b/RUST_DEV.md
index 45b9c78e7..f2abab522 100644
--- a/RUST_DEV.md
+++ b/RUST_DEV.md
@@ -329,29 +329,33 @@ in-memory.
### What goes wrong with the in-memory backend on `--workers N > 1`
-(Historical context — applies only when the operator explicitly
-chooses `CcrBackendConfig::InMemory`.) Each uvicorn worker is a
-separate Python process. Each process holds its own copies of:
-
-1. **`InMemoryCcrStore`** — sharded `DashMap` mapping
- `hash → original_content` for content the compressor replaced with
- `<<ccr:HASH>>` markers.
-2. **`HeadroomProxy._compression_caches`** (`headroom/proxy/server.py:367`)
- — per-session `CompressionCache` dict.
+Each uvicorn worker is a separate Python process. The following state is
+fragmented across workers:
+
+1. **Python `CompressionStore`** — defaults to `InMemoryBackend` (per-process)
+ when `HEADROOM_CCR_BACKEND` is unset. Each worker has its own singleton; CCR
+ markers written on worker A are invisible to worker B. Set
+ `HEADROOM_CCR_BACKEND=sqlite` to use a shared cross-worker store.
+2. **`HeadroomProxy._compression_caches`** (`headroom/proxy/server.py`)
+ — per-session `CompressionCache` dict (instance var, always per-worker).
3. **`HeadroomProxy.session_tracker_store`** — per-session prefix-tracker
- state derived from Anthropic's `cache_read_input_tokens` responses.
-4. **TOIN learner state** — pattern statistics used to bias the compressor.
+ state derived from Anthropic's `cache_read_input_tokens` responses
+ (instance var, always per-worker).
+4. **TOIN learner state** — writes snapshots to `~/.headroom/toin.json` but
+ keeps per-process in-memory state; pattern statistics on one worker are not
+ visible to others until the next disk flush.
When uvicorn round-robins requests across workers, a session whose
turn-1 landed on worker A may have turn-2 land on worker B. Worker B has
zero knowledge of what worker A did, the `<<ccr:HASH>>` marker resolves
to `None`, and the model sees an opaque directive it can't act on.
Switching to `SqliteCcrStore` (default) or `RedisCcrStore` resolves the
-fragmentation directly.
+CCR fragmentation; a sticky-session load balancer resolves all of them.
### Detecting it in the wild
-The proxy emits a `WARNING`-level log line on startup if the configured
-backend is `InMemoryCcrStore` AND `WEB_CONCURRENCY` / uvicorn
-`--workers` is > 1, pointing operators at this section. The other two
-backends never warn — they're the supported multi-worker paths.
+The proxy emits a `WARNING`-level log line on startup when `--workers N > 1`.
+When `HEADROOM_CCR_BACKEND` is unset (default InMemoryBackend), the warning
+includes CCR retrieval failures and suggests setting `HEADROOM_CCR_BACKEND=sqlite`.
+When a cross-worker backend is already configured, the warning covers only the
+remaining per-worker stores (compression cache, prefix tracker, TOIN, CostTracker).
diff --git a/headroom/proxy/server.py b/headroom/proxy/server.py
index c85e3a1a2..8a6867950 100644
--- a/headroom/proxy/server.py
+++ b/headroom/proxy/server.py
@@ -3277,24 +3277,36 @@ def run_server(
app_target: Any
uvicorn_kwargs: dict[str, Any] = {}
if workers > 1:
- # CCR / compression-cache / prefix-tracker / TOIN state are all
- # per-process. Round-robin across workers fragments these caches
- # and produces silent retrieval failures for `Retrieve original:
- # hash=X` markers and avoidable cache busts on the upstream
- # provider. See the "Multi-worker deployment — CCR fragmentation"
- # section in RUST_DEV.md for the full failure modes and the
- # sticky-session workaround.
- logger.warning(
- "Headroom is running with workers=%d. The in-memory CCR store, "
- "compression cache, prefix tracker, TOIN state, and CostTracker are all "
- "per-process; multi-worker deployments produce silent retrieval "
- "failures, avoidable cache busts, and an unstable dashboard 'Proxy $ Saved' "
- "hero tile (each /stats poll hits a different worker's partial total) when "
- "sessions land on different workers. Run --workers 1 (or place a "
- "sticky-session load balancer in front of multiple --workers 1 processes). "
- "See RUST_DEV.md → 'Multi-worker deployment — CCR fragmentation'.",
- workers,
- )
+ # CompressionCache and PrefixTracker are always per-worker instance vars.
+ # Python CompressionStore defaults to InMemoryBackend (per-process), so
+ # CCR markers written on worker A are invisible to worker B unless a
+ # cross-worker backend is configured via HEADROOM_CCR_BACKEND.
+ # See RUST_DEV.md -> "Multi-worker deployment -- CCR fragmentation".
+ if os.environ.get("HEADROOM_CCR_BACKEND", "").strip():
+ logger.warning(
+ "Headroom is running with workers=%d. Compression cache, "
+ "prefix tracker, TOIN state, and CostTracker are all per-process; "
+ "multi-worker deployments produce avoidable cache busts and an "
+ "unstable dashboard 'Proxy $ Saved' hero tile (each /stats poll "
+ "hits a different worker's partial total) when sessions land on "
+ "different workers. Run --workers 1 or place a sticky-session load "
+ "balancer in front of multiple --workers 1 processes. "
+ "See RUST_DEV.md -> 'Multi-worker deployment -- CCR fragmentation'.",
+ workers,
+ )
+ else:
+ logger.warning(
+ "Headroom is running with workers=%d. The in-memory CCR store, "
+ "compression cache, prefix tracker, TOIN state, and CostTracker are all "
+ "per-process; multi-worker deployments produce silent CCR retrieval "
+ "failures, avoidable cache busts, and an unstable dashboard 'Proxy $ Saved' "
+ "hero tile (each /stats poll hits a different worker's partial total) when "
+ "sessions land on different workers. Set HEADROOM_CCR_BACKEND=sqlite for a "
+ "persistent cross-worker CCR store, run --workers 1, or place a "
+ "sticky-session load balancer in front of multiple --workers 1 processes. "
+ "See RUST_DEV.md -> 'Multi-worker deployment -- CCR fragmentation'.",
+ workers,
+ )
os.environ[_MULTI_WORKER_CONFIG_ENV] = json.dumps(_proxy_config_payload(config))
app_target = "headroom.proxy.server:create_app_from_env"
uvicorn_kwargs["factory"] = True