Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(proxy): queue mid-turn user messages on non-Bedrock streaming path (#1377)

Rod Boevb09f0270625a
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c445fc682..0ea1282f9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -35,6 +35,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
* **proxy:** stop discarding a finished compression on very large requests. After the transform pipeline completed, a telemetry-only waste-signal re-parse of the *original* messages ran on the critical path; on huge Claude Code transcripts (~400k tokens) that parse could exceed the Anthropic compression timeout, so the proxy failed open and forwarded the uncompressed request despite "Pipeline complete" logging real savings (`tokens_saved: 0`, `transforms_applied: []`, ~31s latency). Waste-signal detection is now skipped above `MAX_WASTE_SIGNAL_DETECTION_TOKENS` (100k) so the compression result stays on the critical path ([#296](https://github.com/chopratejas/headroom/issues/296)).
* **codex:** retag existing Codex threads when `headroom init` injects the `headroom` provider, so Codex Desktop history stays visible. Codex filters its sidebar/search by the active `model_provider`; the init path set `model_provider = "headroom"` without retagging, so existing native `openai` threads disappeared from the menu (data was never deleted, only hidden). `_ensure_codex_provider` now reconciles thread tags openai→headroom, matching what the install and `wrap` paths already do; `headroom unwrap codex` handles the revert direction ([#961](https://github.com/chopratejas/headroom/issues/961)).
* **install:** stop duplicating the container ENTRYPOINT in the `persistent-docker` runtime command. The published image already runs `headroom proxy` as its ENTRYPOINT, but `build_runtime_command` re-added `headroom proxy` after the image name, so the container ran `headroom proxy headroom proxy --host 0.0.0.0 …` and Click aborted with "Got unexpected extra arguments (headroom proxy)" — the deployment never became ready and rollback left nothing running. The runtime command now appends only the proxy flags ([#833](https://github.com/chopratejas/headroom/issues/833)).
+* **proxy:** queue mid-turn user messages on non-Bedrock streaming path instead of silently dropping them — closes [#902](https://github.com/headroomlabs-ai/headroom/issues/902).
* **proxy:** add `--protect-tool-results` / `HEADROOM_PROTECT_TOOL_RESULTS` to prevent lossy compression of exact-output tool results (e.g. `Bash cat`/`grep` results) — closes [#1307](https://github.com/headroomlabs-ai/headroom/issues/1307).
* **cli:** add `--rpm`/`--tpm` and `HEADROOM_RPM`/`HEADROOM_TPM` to the Click proxy command for rate-limit parity with the legacy CLI -- closes [#1350](https://github.com/headroomlabs-ai/headroom/issues/1350) (Problem 1).
* **proxy:** register `ToolResultInterceptorTransform` in explicit transforms list when `HEADROOM_INTERCEPT_ENABLED` is set — closes [#829](https://github.com/headroomlabs-ai/headroom/issues/829).
diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.py
index a88cc6d91..500840da9 100644
--- a/headroom/proxy/handlers/anthropic.py
+++ b/headroom/proxy/handlers/anthropic.py
@@ -2140,6 +2140,15 @@ class AnthropicHandlerMixin:
metadata={"path": pipeline_path, "stream": True},
)
await _finalize_pre_upstream()
+ session_key = self._get_session_key(
+ body,
+ session_header=request.headers.get("x-headroom-session-id"),
+ )
+ if session_key in self._active_streams:
+ from fastapi.responses import JSONResponse
+
+ queued = self._queue_mid_turn_message(session_key, body)
+ return JSONResponse(content=queued, status_code=202)
return await self._stream_response(
url,
headers,
@@ -2162,6 +2171,7 @@ class AnthropicHandlerMixin:
mutation_reasons=body_mutation_tracker.reasons,
memory_request_ctx=memory_request_ctx,
outcome_provider=provider_name,
+ session_key=session_key,
)
else:
async with stage_timer.measure("upstream_connect"):
diff --git a/headroom/proxy/handlers/streaming.py b/headroom/proxy/handlers/streaming.py
index 6ad688fa8..ab0c6a72d 100644
--- a/headroom/proxy/handlers/streaming.py
+++ b/headroom/proxy/handlers/streaming.py
@@ -59,6 +59,51 @@ def _parse_completion_tokens_from_sse_chunk(chunk_bytes: bytes) -> int | None:
class StreamingMixin:
"""Mixin providing streaming response methods for HeadroomProxy."""
+ _mid_turn_queues: dict[str, asyncio.Queue] = {}
+ _active_streams: set[str] = set()
+
+ @staticmethod
+ def _get_session_key(body: dict, session_header: str | None = None) -> str:
+ """Return session identity from an explicit header or a body-derived hash.
+
+ Fallback mirrors prefix_tracker.compute_session_id: md5(model:system[:500]).
+ """
+ if session_header:
+ return session_header
+ import hashlib
+
+ system = body.get("system", "")
+ if isinstance(system, list):
+ for block in system:
+ if isinstance(block, dict) and block.get("type") == "text":
+ system = block.get("text", "")
+ break
+ else:
+ system = ""
+ system_content = str(system)[:500]
+ key = f"{body.get('model', '')}:{system_content}"
+ return hashlib.md5(key.encode()).hexdigest()[:16]
+
+ def _queue_mid_turn_message(self, session_key: str, body: dict) -> dict:
+ """Queue a mid-turn message and return a 202 response."""
+ if session_key not in self._mid_turn_queues:
+ self._mid_turn_queues[session_key] = asyncio.Queue()
+ self._mid_turn_queues[session_key].put_nowait(body)
+ return {"status": 202, "event": "headroom_queued"}
+
+ def _cleanup_mid_turn_stream(
+ self, session_key: str, *, drain_pending_messages: bool = False
+ ) -> list[dict]:
+ """Clear active mid-turn state, optionally returning queued messages."""
+ self._active_streams.discard(session_key)
+ queue = self._mid_turn_queues.pop(session_key, None)
+ if not drain_pending_messages or queue is None or queue.empty():
+ return []
+ pending_messages: list[dict] = []
+ while not queue.empty():
+ pending_messages.append(queue.get_nowait())
+ return pending_messages
+
@staticmethod
def _extract_anthropic_cache_ttl_metrics(usage: dict[str, Any] | None) -> tuple[int, int]:
"""Extract observed Anthropic cache-write TTL bucket usage."""
@@ -837,6 +882,7 @@ class StreamingMixin:
memory_request_ctx: Any | None = None,
outcome_provider: str | None = None,
waste_signals: dict[str, int] | None = None,
+ session_key: str | None = None,
) -> Response | StreamingResponse:
"""Stream response with metrics tracking and memory tool handling.
@@ -851,6 +897,9 @@ class StreamingMixin:
"""
from fastapi.responses import Response, StreamingResponse
+ session_key = session_key or self._get_session_key(body)
+ self._active_streams.add(session_key)
+
from headroom.proxy.helpers import MAX_SSE_BUFFER_SIZE
# Identify the harness (codex / claude-code / aider / cursor /
@@ -1016,6 +1065,7 @@ class StreamingMixin:
}
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
+ self._cleanup_mid_turn_stream(session_key)
return StreamingResponse(_error_gen(), media_type="text/event-stream")
# Capture Codex rate-limit window data from the upstream response
@@ -1113,6 +1163,7 @@ class StreamingMixin:
client=client,
waste_signals=waste_signals,
)
+ self._cleanup_mid_turn_stream(session_key)
return Response(
content=error_content,
status_code=upstream_response.status_code,
@@ -1148,6 +1199,8 @@ class StreamingMixin:
# corrupted strings.
full_sse_bytes = bytearray()
parsed_response = None # Set by memory block; used by CCR + prefix tracker
+ completed_normally = False
+ pending_messages: list[dict] = []
try:
async with contextlib.aclosing(upstream_response) as response:
@@ -1318,6 +1371,7 @@ class StreamingMixin:
status_code=upstream_response.status_code,
metadata={"total_bytes": stream_state["total_bytes"]},
)
+ completed_normally = True
except (httpx.ConnectError, httpx.ConnectTimeout, httpx.PoolTimeout) as e:
logger.error(f"[{request_id}] Connection error to upstream API: {e}")
@@ -1341,6 +1395,10 @@ class StreamingMixin:
}
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
finally:
+ pending_messages = self._cleanup_mid_turn_stream(
+ session_key,
+ drain_pending_messages=completed_normally,
+ )
# PR-A8 / P1-8: best-effort decode for downstream
# finalization. This runs in `finally` so it must not
# raise — if the upstream sent invalid bytes mid-stream
@@ -1384,6 +1442,11 @@ class StreamingMixin:
client=client,
waste_signals=waste_signals,
)
+ if pending_messages:
+ pending_event = json.dumps(
+ {"type": "headroom_pending_messages", "messages": pending_messages}
+ )
+ yield f"event: headroom_pending_messages\ndata: {pending_event}\n\n".encode()
return StreamingResponse(
generate(),
diff --git a/tests/test_mid_turn_steering.py b/tests/test_mid_turn_steering.py
new file mode 100644
index 000000000..fae01a5c9
--- /dev/null
+++ b/tests/test_mid_turn_steering.py
@@ -0,0 +1,175 @@
+import asyncio
+from unittest.mock import AsyncMock, MagicMock
+
+import httpx
+import pytest
+
+from headroom.proxy.server import HeadroomProxy
+
+
+class TestMidTurnSteering:
+ def test_mid_turn_queue_exists_on_streaming_mixin(self):
+ """StreamingMixin has _mid_turn_queues class attribute after the fix."""
+ from headroom.proxy.handlers.streaming import StreamingMixin
+
+ assert hasattr(StreamingMixin, "_mid_turn_queues")
+ assert hasattr(StreamingMixin, "_active_streams")
+
+ def test_mid_turn_message_queued_when_stream_active(self):
+ """When a session has an active stream, mid-turn messages are queued."""
+ from headroom.proxy.handlers.streaming import StreamingMixin
+
+ mixin = StreamingMixin()
+ session_key = "test-session-123"
+ mixin._active_streams.add(session_key)
+ body = {"messages": [{"role": "user", "content": "follow-up"}]}
+ result = mixin._queue_mid_turn_message(session_key, body)
+ assert result["status"] == 202
+ assert result["event"] == "headroom_queued"
+ assert not mixin._mid_turn_queues[session_key].empty()
+ queued = mixin._mid_turn_queues[session_key].get_nowait()
+ assert queued == body
+ # Cleanup
+ mixin._active_streams.discard(session_key)
+ del mixin._mid_turn_queues[session_key]
+
+ def test_no_queue_when_no_prior_stream(self):
+ """When no stream is active, _mid_turn_queues stays empty for the session."""
+ from headroom.proxy.handlers.streaming import StreamingMixin
+
+ mixin = StreamingMixin()
+ session_key = "inactive-session"
+ assert session_key not in mixin._active_streams
+ assert session_key not in mixin._mid_turn_queues
+
+ def _create_mock_proxy(self):
+ proxy = object.__new__(HeadroomProxy)
+ proxy.http_client = MagicMock(spec=httpx.AsyncClient)
+ proxy._config = MagicMock()
+ proxy._config.memory_enabled = False
+ proxy._config.ccr_inject_tool = False
+ proxy._config.retry_max_attempts = 1
+ proxy._config.retry_base_delay_ms = 0
+ proxy._config.retry_max_delay_ms = 0
+ proxy.config = proxy._config
+ proxy.memory_handler = None
+ proxy._parse_sse_usage_from_buffer = MagicMock(return_value=None)
+ proxy._finalize_stream_response = AsyncMock(return_value=None)
+ return proxy
+
+ @staticmethod
+ def _create_mock_upstream_response(
+ chunks: list[bytes], *, terminal_exception: BaseException | None = None
+ ):
+ mock_response = AsyncMock()
+ mock_response.headers = httpx.Headers({"content-type": "text/event-stream"})
+ mock_response.status_code = 200
+
+ async def aiter_bytes():
+ for chunk in chunks:
+ yield chunk
+ if terminal_exception is not None:
+ raise terminal_exception
+
+ mock_response.aiter_bytes = aiter_bytes
+ mock_response.aclose = AsyncMock()
+ return mock_response
+
+ @pytest.mark.asyncio
+ async def test_mid_turn_stream_cancellation_clears_active_session_and_queue(self):
+ proxy = self._create_mock_proxy()
+ session_key = "cancelled-session"
+ mock_response = self._create_mock_upstream_response(
+ [
+ b'event: message_start\ndata: {"type":"message_start"}\n\n',
+ ],
+ terminal_exception=asyncio.CancelledError(),
+ )
+
+ proxy.http_client.build_request = MagicMock(return_value=MagicMock())
+ proxy.http_client.send = AsyncMock(return_value=mock_response)
+
+ result = await proxy._stream_response(
+ url="https://api.anthropic.com/v1/messages",
+ headers={"x-api-key": "sk-test", "x-headroom-session-id": session_key},
+ body={
+ "model": "claude-sonnet-4-20250514",
+ "max_tokens": 100,
+ "stream": True,
+ "messages": [{"role": "user", "content": "hi"}],
+ },
+ provider="anthropic",
+ model="claude-sonnet-4-20250514",
+ request_id="test-cancelled",
+ original_tokens=10,
+ optimized_tokens=10,
+ tokens_saved=0,
+ transforms_applied=[],
+ tags={},
+ optimization_latency=0.0,
+ session_key=session_key,
+ )
+ proxy._queue_mid_turn_message(
+ session_key,
+ {"messages": [{"role": "user", "content": "follow-up"}]},
+ )
+
+ try:
+ with pytest.raises(asyncio.CancelledError):
+ async for _chunk in result.body_iterator:
+ pass
+ assert session_key not in proxy._active_streams
+ assert session_key not in proxy._mid_turn_queues
+ mock_response.aclose.assert_awaited_once()
+ finally:
+ proxy._active_streams.discard(session_key)
+ proxy._mid_turn_queues.pop(session_key, None)
+
+ @pytest.mark.asyncio
+ async def test_mid_turn_stream_exception_clears_active_session_and_queue(self):
+ proxy = self._create_mock_proxy()
+ session_key = "errored-session"
+ mock_response = self._create_mock_upstream_response(
+ [
+ b'event: message_start\ndata: {"type":"message_start"}\n\n',
+ ],
+ terminal_exception=RuntimeError("stream exploded"),
+ )
+
+ proxy.http_client.build_request = MagicMock(return_value=MagicMock())
+ proxy.http_client.send = AsyncMock(return_value=mock_response)
+
+ result = await proxy._stream_response(
+ url="https://api.anthropic.com/v1/messages",
+ headers={"x-api-key": "sk-test", "x-headroom-session-id": session_key},
+ body={
+ "model": "claude-sonnet-4-20250514",
+ "max_tokens": 100,
+ "stream": True,
+ "messages": [{"role": "user", "content": "hi"}],
+ },
+ provider="anthropic",
+ model="claude-sonnet-4-20250514",
+ request_id="test-errored",
+ original_tokens=10,
+ optimized_tokens=10,
+ tokens_saved=0,
+ transforms_applied=[],
+ tags={},
+ optimization_latency=0.0,
+ session_key=session_key,
+ )
+ proxy._queue_mid_turn_message(
+ session_key,
+ {"messages": [{"role": "user", "content": "follow-up"}]},
+ )
+
+ try:
+ chunks = [chunk async for chunk in result.body_iterator]
+ assert any(b"event: error" in chunk for chunk in chunks)
+ assert session_key not in proxy._active_streams
+ assert session_key not in proxy._mid_turn_queues
+ mock_response.aclose.assert_awaited_once()
+ finally:
+ proxy._active_streams.discard(session_key)
+ proxy._mid_turn_queues.pop(session_key, None)