Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(learn): don't desync verbosity pairing on empty assistant turns (#2123)
Abhay Singhdef2f9a7286a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 467829582..b2ac2e465 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -133,6 +133,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0* **tokenizers:** resolve HuggingFace tokenizer names by the most-specific prefix. `get_tokenizer_name` scanned `MODEL_TO_TOKENIZER` in dict-insertion order and returned the first key the model merely starts with, so a short family key shadowed a more-specific one — `qwen2-7b-instruct` matched `qwen` before `qwen2`/`qwen2-7b` and resolved to the Qwen1 tokenizer (a different vocabulary, hence wrong token counts); `qwen2.5-*` and `deepseek-v2.x` were mis-resolved the same way. It now picks the longest matching prefix, mirroring the order-dependent-prefix guard the sibling tiktoken `get_encoding_for_model` already documents.* **pricing:** map retired `claude-3-sonnet-20240229` to a Sonnet-tier price instead of Haiku. When LiteLLM's cost DB lacks the retired model, resolution falls through to `MODEL_ALIASES`, which pointed Claude 3 Sonnet (a $3/$15-per-1M model) at `claude-3-haiku-20240307` ($0.25/$1.25) — a different tier that underpriced every cost/savings figure for that model ~12x on both input and output. It now aliases to `claude-sonnet-4-20250514`, the same-price target the other retired-Sonnet aliases already use.* **cache/semantic:** don't evict an unrelated entry when re-storing a key that is already cached. `SemanticCache.put` ran its at-capacity eviction loop before computing the entry's key, so overwriting a key that was already present (a duplicate or retried store) still evicted the LRU-oldest distinct entry even though an in-place update grows nothing. That silently dropped a live entry and turned a later lookup for it into a false cache miss. The key is now computed first and the eviction loop only runs when the key is genuinely new (mirroring `CompressionCache.store_compressed`, which deletes-then-inserts).+* **learn:** keep `verbosity._ordered_events` in lockstep with `_parse_session` on empty assistant turns. `_parse_session` creates a `_Response` only when an assistant message has content (`words > 0 or out_tok > 0`), but `_ordered_events` consumed a response slot for *every* assistant line. An empty assistant turn (e.g. a pure `tool_use` turn with no usage) therefore shifted the `responses` list out of alignment, so a later human reply was paired with a future-timestamped response, the read gap went negative, and a spurious `fast_skip` was recorded — inflating `fast_skip_rate` and skewing the recommended verbosity level. `_ordered_events` now applies the same `words > 0 or out_tok > 0` guard before consuming a response, matching the filter the user side already mirrors.* **wrap/claude:** don't raise `UnboundLocalError` in the cleanup `finally` when the proxy fails to start. `claude()` referenced `_wrap_settings_path` in its `finally` block but only assigned it inside the `try`, after `_ensure_proxy` (which raises on port exhaustion or a failed proxy start). An early failure therefore made the `finally` raise `UnboundLocalError` — replacing the real error with a raw traceback and, because the `finally` aborted before `cleanup()`, skipping proxy cleanup and wrap-marker clearing. `_wrap_settings_path` is now bound before the `try` alongside the other cleanup holders (`proxy_holder`, `_saved_base_url`, …), so the `finally` is always safe.* **wrap/codex:** export the detected custom upstream base URL so Codex actually routes to it. `_inject_codex_provider_config` detects an OpenAI-compatible gateway declared in `~/.codex/config.toml` and writes an `X-Headroom-Base-Url` header mapped to `HEADROOM_CODEX_UPSTREAM_BASE_URL`, and it returns that URL for the caller to export. But `_prepare_codex_wrap_state` discarded the return and `_run_codex_wrap` never set the env var, so Codex emitted no header, the proxy fell back to `api.openai.com`, and the user's gateway key was sent to OpenAI (which rejects it). This restores the wiring a later refactor dropped: `_prepare_codex_wrap_state` now returns the URL and `_run_codex_wrap` exports it into the launch env when set (a user-provided value still wins) (regression of [#1614](https://github.com/chopratejas/headroom/issues/1614)).* **cache:** normalize embeddings before the semantic dynamic-content similarity check. `SemanticDetector` scored sentences with `np.dot` against exemplar embeddings and compared the result to `semantic_threshold` (a 0-1 cosine value), but `sentence_transformers.encode(..., convert_to_numpy=True)` does not normalize, so the dot product was an unbounded inner product (vector norms ~5-15) rather than a cosine similarity. Nearly every sentence cleared the 0.7 threshold, so the semantic tier flagged almost all text as dynamic and stripped static content, busting the cache it is meant to protect (a standalone repro scores an unrelated sentence, true cosine ~0.1, at a raw dot of ~9). Both `encode` calls now pass `normalize_embeddings=True`, matching the siblings in `prediction/feature_extractor.py` and `memory/adapters/embedders.py`, so the dot product is a true cosine in [-1, 1].diff --git a/headroom/learn/verbosity.py b/headroom/learn/verbosity.pyindex 4d428c922..a67024fae 100644--- a/headroom/learn/verbosity.py+++ b/headroom/learn/verbosity.py@@ -294,9 +294,21 @@ def _ordered_events(path: Path) -> list[tuple[float | None, str, _Response | _Huexcept json.JSONDecodeError:continueltype = d.get("type")- if ltype == "assistant" and ri < len(responses):- out.append((responses[ri].ts, "assistant", responses[ri]))- ri += 1+ if ltype == "assistant":+ # Mirror the _parse_session filter (a _Response is only created when+ # `words > 0 or out_tok > 0`). Without the same guard here, an empty+ # assistant line (e.g. a pure tool_use turn with no usage) consumes a+ # response slot that belongs to a later real response, desyncing the+ # two lists so a following human reply is paired with the wrong+ # (future-timestamped) response -> a negative gap -> a spurious+ # fast_skip.+ msg = d.get("message", {}) if isinstance(d.get("message"), dict) else {}+ words, _ = _assistant_words_and_text(msg.get("content", []))+ usage = msg.get("usage", {}) if isinstance(msg.get("usage"), dict) else {}+ out_tok = usage.get("output_tokens", 0)+ if (words > 0 or out_tok > 0) and ri < len(responses):+ out.append((responses[ri].ts, "assistant", responses[ri]))+ ri += 1elif ltype == "user":msg = d.get("message", {}) if isinstance(d.get("message"), dict) else {}text = _human_text(msg.get("content"))diff --git a/tests/test_verbosity_learn.py b/tests/test_verbosity_learn.pyindex 538a3569f..0be9a5242 100644--- a/tests/test_verbosity_learn.py+++ b/tests/test_verbosity_learn.py@@ -55,6 +55,22 @@ def _tool_result(*, ts: str, content: str = "ok") -> dict:}+def _empty_assistant(*, ts: str) -> dict:+ """A pure tool_use assistant turn with no text and no output tokens.++ `_parse_session` creates no `_Response` for it (words == 0 and out_tok == 0).+ """+ return {+ "type": "assistant",+ "timestamp": ts,+ "message": {+ "model": "claude-opus-4-8",+ "content": [{"type": "tool_use", "id": "t1", "name": "Bash", "input": {}}],+ "usage": {"input_tokens": 100, "output_tokens": 0},+ },+ }++LONG = " ".join(["word"] * 400) # well above the long-output floor@@ -103,6 +119,28 @@ class TestSignalExtraction:assert sig.skip_eligible == 1assert sig.fast_skips == 0+ def test_empty_assistant_message_does_not_desync_fast_skip(self, tmp_path):+ # An empty assistant turn (pure tool_use, no output) creates no response+ # at parse time, so _ordered_events must not consume a response slot for+ # it. Otherwise a later real answer's slot is consumed early, the slow+ # reply below is paired with a future-timestamped response, the gap goes+ # negative, and a spurious fast_skip is recorded.+ p = _write_session(+ tmp_path,+ "s",+ [+ _user("explain", ts="2026-01-01T00:00:00Z"),+ _empty_assistant(ts="2026-01-01T00:00:00Z"),+ _assistant(LONG, ts="2026-01-01T00:00:01Z"), # real answer #1+ # +199s: a slow, considered reply — NOT a fast skip.+ _user("here is my careful follow-up", ts="2026-01-01T00:03:20Z"),+ _assistant(LONG, ts="2026-01-01T00:03:21Z"), # real answer #2+ _user("thanks", ts="2026-01-01T00:07:00Z"),+ ],+ )+ sig, _ = extract_signals([p])+ assert sig.fast_skips == 0+def test_short_answer_not_skip_eligible(self, tmp_path):p = _write_session(tmp_path,