Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(proxy/memory): capture user text blocks for the retrieval query (#2064)

Abhay Singhf542b7041367
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d5ce2003b..c9834eebe 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -108,6 +108,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
* **tokenizers:** recurse into a native `tool_result` whose content is a list of blocks instead of JSON-serializing it. `_count_content_parts` counted a `tool_result` with list content via `_count_serialized` (json.dumps + sample), so a base64 image nested in a tool result (computer-use / MCP screenshot tools) was priced as text — a ~50-200x overcount (a ~200KB screenshot read as ~70K tokens instead of ~1600). It now recurses into the nested blocks, matching the sibling Strands `toolResult` branch, so the image is priced structurally. The overcount made a single screenshot appear to blow past the model's context window and triggered unnecessary/over-aggressive compression.
* **tokenizers:** price dense scripts (CJK/Kana/Hangul) in the fixed-ratio estimator path. `EstimatingTokenCounter.count_text` applied the CJK correction only on the auto-detect path; its fixed-ratio early return divided by the Latin ratio with no adjustment. The registry builds every provider-calibrated counter with a fixed ratio (Anthropic 3.5, Google 4.0, Cohere 4.0, Moonshot 3.1), and the Anthropic/Gemini proxy handlers count via `get_tokenizer(model).count_messages`, so a CJK-heavy context read as ~40-55% of its true token size — it could fall under the size/backpressure gates and skip compression, and every `x-headroom-tokens-before` metric for CJK traffic was materially wrong. The fixed-ratio path now applies the same dense-script split.
* **ccr:** don't crash `parse_tool_call` on a CCR tool call whose arguments aren't an object. For the OpenAI/`openai_responses` shape the arguments are `json.loads`-decoded and only `JSONDecodeError` was caught, so a model that emitted `arguments='[]'`/`'"abc"'`/`'123'` (decoding to a list/str/number) — or a non-dict Anthropic `input` — reached `input_data.get("hash")` and raised `AttributeError`; a null `arguments` raised an uncaught `TypeError` from `json.loads(None)`. Both are now handled: the decode also catches `TypeError`, and a non-dict `input_data` returns `None` (not a valid CCR call) instead of crashing CCR response processing.
+* **proxy/memory:** capture the user's prompt from Anthropic text blocks when building the memory-retrieval query. `extract_memory_query_sources` only recorded `latest_user` when a user message's `content` was a plain string; for the standard Anthropic `/v1/messages` shape (`content=[{"type":"text","text":...}]`, used by Claude Code) it routed into the tool-result extractor and never read the `text` blocks — so the actual question was discarded. On a first turn the embedding query was then empty and memory injection was silently skipped entirely; with history it keyed on stale assistant/tool context instead of the user's ask. The user turn's `text` blocks are now captured.
* **memory/sqlite:** stop `SQLiteMemoryStore.query` from emitting `OFFSET` without a `LIMIT`. The query builder appended `LIMIT` only when `filter.limit is not None` and `OFFSET` independently when `filter.offset > 0`, but SQLite accepts `OFFSET` only as part of a `LIMIT` clause — so a `MemoryFilter(offset=N)` with no limit produced `... OFFSET ?` and crashed with `sqlite3.OperationalError: near "OFFSET": syntax error`. An offset-without-limit now emits SQLite's unbounded `LIMIT -1` so pagination works.
* **mcp/codex:** don't corrupt an unparseable or non-table `config.toml` on register. `CodexRegistrar.register_server` only guarded against clobbering a user-managed entry when `get_server` returned one, but `get_server` returns `None` both for an unparseable TOML file and for an `mcp_servers`/`mcp_servers.<name>` that is present but not a table. In those cases `register_server` fell through to `_write_block`, which blindly appended a `[mcp_servers.<name>]` table — appending into an unparseable file, or creating a duplicate `[mcp_servers.headroom]` key alongside a non-table entry (e.g. `headroom = "..."`), which `tomllib`/codex then reject, destroying a previously-valid config. It now refuses (`FAILED`) and leaves the file untouched, mirroring the claude (#1660) and opencode (#1661) guards.
* **proxy/vertex:** route Vertex `publisher=google` (Gemini) requests to the region matching the request path. `vertex_generate_content`, `vertex_stream_generate_content`, and `vertex_count_tokens` discarded the path's `location` and forwarded to the single fixed host from `_api_target(proxy, "vertex")` (default `us-central1`), instead of the region-aware `_vertex_target_for_location` the sibling Anthropic `rawPredict` route already uses. So a request to `.../locations/europe-west1/publishers/google/...` was sent to a `us-central1` host, which Vertex rejects on the region/host mismatch. The three google routes now derive the host from the request's `location` (operator-pinned upstreams are still honored).
diff --git a/headroom/proxy/memory_query_policy.py b/headroom/proxy/memory_query_policy.py
index 263605ebc..8d765e247 100644
--- a/headroom/proxy/memory_query_policy.py
+++ b/headroom/proxy/memory_query_policy.py
@@ -59,6 +59,18 @@ def extract_memory_query_sources(
tool_outputs=tool_outputs,
lookback_tools=lookback_tools,
)
+ if not latest_user:
+ # Anthropic user turns carry the actual prompt as text blocks
+ # ({"type":"text","text":...}), not a plain string. Capture it
+ # so the memory retrieval query keys on the user's question and
+ # not just any tool_result blocks in the same turn.
+ user_text = "\n".join(
+ b.get("text", "")
+ for b in content
+ if isinstance(b, dict) and b.get("type") == "text"
+ ).strip()
+ if user_text:
+ latest_user = user_text
elif isinstance(content, str) and not latest_user:
latest_user = content
diff --git a/tests/test_memory_query_policy.py b/tests/test_memory_query_policy.py
index ea88617d7..cc80d6a62 100644
--- a/tests/test_memory_query_policy.py
+++ b/tests/test_memory_query_policy.py
@@ -54,3 +54,35 @@ def test_extract_sources_handles_anthropic_tool_result_without_user_text() -> No
assert user_text == "real user"
assert tool_outputs == ("nested",)
assert assistant_turns == ()
+
+
+def test_extract_sources_captures_anthropic_user_text_blocks() -> None:
+ """Anthropic user turns carry the prompt as text blocks (the standard Claude
+ Code shape). The user's question must be captured — not dropped — so memory
+ retrieval keys on it."""
+ messages = [
+ {"role": "user", "content": [{"type": "text", "text": "help me refactor auth"}]},
+ ]
+
+ user_text, _tool_outputs, _assistant_turns = extract_memory_query_sources(messages)
+
+ assert user_text == "help me refactor auth"
+
+
+def test_extract_sources_captures_user_text_alongside_tool_result() -> None:
+ """A user turn mixing a tool_result and a text block yields both: the text as
+ the user query and the tool output as context."""
+ messages = [
+ {
+ "role": "user",
+ "content": [
+ {"type": "tool_result", "content": "exit 0"},
+ {"type": "text", "text": "did the tests pass?"},
+ ],
+ },
+ ]
+
+ user_text, tool_outputs, _assistant_turns = extract_memory_query_sources(messages)
+
+ assert user_text == "did the tests pass?"
+ assert tool_outputs == ("exit 0",)