Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(proxy): make CCR multi-worker warning conditional on backend (#770)
Patrick Ad76a7296df12
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7545d2140..3414bdb77 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -78,6 +78,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Bug Fixes* **codex:** respect `CODEX_HOME` when `headroom wrap codex` writes provider, MCP, memory, backup, and global `AGENTS.md` config, and warn when `unwrap codex` may be looking at the default Codex home because `CODEX_HOME` is unset.+* **proxy:** multi-worker CCR warning is now conditional on backend — when `HEADROOM_CCR_BACKEND` is unset (default `InMemoryBackend`, per-process), the startup warning includes CCR retrieval failures and suggests `HEADROOM_CCR_BACKEND=sqlite`; when a cross-worker backend is already configured, the warning covers only the remaining per-worker stores (compression cache, prefix tracker, TOIN, CostTracker). Updated `RUST_DEV.md` to accurately document Python `CompressionStore` as per-process by default.* **deps:** move `gunicorn` to `[proxy-prod]` extra with `sys_platform != 'win32'` guard; removed from `[proxy]` to avoid forcing a Unix-only package on dev, CI, and Windows users ([#537](https://github.com/chopratejas/headroom/pull/537))* **startup:** suppress proxy startup log noise -- litellm banner, trafilatura parse errors, HuggingFace Hub unauthenticated warnings, tiktoken fallback warning, and httpx INFO lines from sentence_transformers HEAD checks. Affected files: `headroom/providers/litellm.py`, `headroom/transforms/html_extractor.py`, `headroom/memory/adapters/embedders.py`, `headroom/providers/anthropic.py`, `headroom/providers/registry.py`, `headroom/image/onnx_router.py`, `headroom/transforms/kompress_compressor.py`.diff --git a/RUST_DEV.md b/RUST_DEV.mdindex 45b9c78e7..f2abab522 100644--- a/RUST_DEV.md+++ b/RUST_DEV.md@@ -329,29 +329,33 @@ in-memory.### What goes wrong with the in-memory backend on `--workers N > 1`-(Historical context — applies only when the operator explicitly-chooses `CcrBackendConfig::InMemory`.) Each uvicorn worker is a-separate Python process. Each process holds its own copies of:--1. **`InMemoryCcrStore`** — sharded `DashMap` mapping- `hash → original_content` for content the compressor replaced with- `<<ccr:HASH>>` markers.-2. **`HeadroomProxy._compression_caches`** (`headroom/proxy/server.py:367`)- — per-session `CompressionCache` dict.+Each uvicorn worker is a separate Python process. The following state is+fragmented across workers:++1. **Python `CompressionStore`** — defaults to `InMemoryBackend` (per-process)+ when `HEADROOM_CCR_BACKEND` is unset. Each worker has its own singleton; CCR+ markers written on worker A are invisible to worker B. Set+ `HEADROOM_CCR_BACKEND=sqlite` to use a shared cross-worker store.+2. **`HeadroomProxy._compression_caches`** (`headroom/proxy/server.py`)+ — per-session `CompressionCache` dict (instance var, always per-worker).3. **`HeadroomProxy.session_tracker_store`** — per-session prefix-tracker- state derived from Anthropic's `cache_read_input_tokens` responses.-4. **TOIN learner state** — pattern statistics used to bias the compressor.+ state derived from Anthropic's `cache_read_input_tokens` responses+ (instance var, always per-worker).+4. **TOIN learner state** — writes snapshots to `~/.headroom/toin.json` but+ keeps per-process in-memory state; pattern statistics on one worker are not+ visible to others until the next disk flush.When uvicorn round-robins requests across workers, a session whoseturn-1 landed on worker A may have turn-2 land on worker B. Worker B haszero knowledge of what worker A did, the `<<ccr:HASH>>` marker resolvesto `None`, and the model sees an opaque directive it can't act on.Switching to `SqliteCcrStore` (default) or `RedisCcrStore` resolves the-fragmentation directly.+CCR fragmentation; a sticky-session load balancer resolves all of them.### Detecting it in the wild-The proxy emits a `WARNING`-level log line on startup if the configured-backend is `InMemoryCcrStore` AND `WEB_CONCURRENCY` / uvicorn-`--workers` is > 1, pointing operators at this section. The other two-backends never warn — they're the supported multi-worker paths.+The proxy emits a `WARNING`-level log line on startup when `--workers N > 1`.+When `HEADROOM_CCR_BACKEND` is unset (default InMemoryBackend), the warning+includes CCR retrieval failures and suggests setting `HEADROOM_CCR_BACKEND=sqlite`.+When a cross-worker backend is already configured, the warning covers only the+remaining per-worker stores (compression cache, prefix tracker, TOIN, CostTracker).diff --git a/headroom/proxy/server.py b/headroom/proxy/server.pyindex c85e3a1a2..8a6867950 100644--- a/headroom/proxy/server.py+++ b/headroom/proxy/server.py@@ -3277,24 +3277,36 @@ def run_server(app_target: Anyuvicorn_kwargs: dict[str, Any] = {}if workers > 1:- # CCR / compression-cache / prefix-tracker / TOIN state are all- # per-process. Round-robin across workers fragments these caches- # and produces silent retrieval failures for `Retrieve original:- # hash=X` markers and avoidable cache busts on the upstream- # provider. See the "Multi-worker deployment — CCR fragmentation"- # section in RUST_DEV.md for the full failure modes and the- # sticky-session workaround.- logger.warning(- "Headroom is running with workers=%d. The in-memory CCR store, "- "compression cache, prefix tracker, TOIN state, and CostTracker are all "- "per-process; multi-worker deployments produce silent retrieval "- "failures, avoidable cache busts, and an unstable dashboard 'Proxy $ Saved' "- "hero tile (each /stats poll hits a different worker's partial total) when "- "sessions land on different workers. Run --workers 1 (or place a "- "sticky-session load balancer in front of multiple --workers 1 processes). "- "See RUST_DEV.md → 'Multi-worker deployment — CCR fragmentation'.",- workers,- )+ # CompressionCache and PrefixTracker are always per-worker instance vars.+ # Python CompressionStore defaults to InMemoryBackend (per-process), so+ # CCR markers written on worker A are invisible to worker B unless a+ # cross-worker backend is configured via HEADROOM_CCR_BACKEND.+ # See RUST_DEV.md -> "Multi-worker deployment -- CCR fragmentation".+ if os.environ.get("HEADROOM_CCR_BACKEND", "").strip():+ logger.warning(+ "Headroom is running with workers=%d. Compression cache, "+ "prefix tracker, TOIN state, and CostTracker are all per-process; "+ "multi-worker deployments produce avoidable cache busts and an "+ "unstable dashboard 'Proxy $ Saved' hero tile (each /stats poll "+ "hits a different worker's partial total) when sessions land on "+ "different workers. Run --workers 1 or place a sticky-session load "+ "balancer in front of multiple --workers 1 processes. "+ "See RUST_DEV.md -> 'Multi-worker deployment -- CCR fragmentation'.",+ workers,+ )+ else:+ logger.warning(+ "Headroom is running with workers=%d. The in-memory CCR store, "+ "compression cache, prefix tracker, TOIN state, and CostTracker are all "+ "per-process; multi-worker deployments produce silent CCR retrieval "+ "failures, avoidable cache busts, and an unstable dashboard 'Proxy $ Saved' "+ "hero tile (each /stats poll hits a different worker's partial total) when "+ "sessions land on different workers. Set HEADROOM_CCR_BACKEND=sqlite for a "+ "persistent cross-worker CCR store, run --workers 1, or place a "+ "sticky-session load balancer in front of multiple --workers 1 processes. "+ "See RUST_DEV.md -> 'Multi-worker deployment -- CCR fragmentation'.",+ workers,+ )os.environ[_MULTI_WORKER_CONFIG_ENV] = json.dumps(_proxy_config_payload(config))app_target = "headroom.proxy.server:create_app_from_env"uvicorn_kwargs["factory"] = True