Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(mcp): reap orphaned mcp serve on client death (#2226)

Vsevolod Rychkov7a5d8a7ace2c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 63239d167..3a0f2d09f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **install:** `headroom install apply --env KEY=VALUE` (repeatable) passes environment variables into supervised runners (macOS launchd, Linux systemd/cron, Windows services/tasks). These runners previously started with a bare environment and did not inherit the interactive shell's exports — e.g. a custom `HEADROOM_WORKSPACE_DIR` never reached the supervised process, so `headroom install agent run` looked for its manifest in the wrong location and failed outright even though `install apply` itself succeeded. `--env` values are merged into `DeploymentManifest.base_env` last, so they can override auto-derived defaults, and are threaded into the generated `run-headroom.sh`/`ensure-headroom.sh` (and Windows equivalents) as `export`/`$env:` lines before the `exec`.
### Fixed
+- **mcp:** reap orphaned `headroom mcp serve` processes when the launching client dies. An MCP stdio server relies on stdin EOF to shut down, but an abrupt client `SIGKILL` leaves the SDK's blocking stdin-reader thread wedged, so `server.run()` never returns; the process is reparented to init/launchd (`ppid == 1`) and lingers, pinning one Python interpreter + tree-sitter grammars per dead session (observed 3+ simultaneously). `run_stdio()` now runs a parent-death watchdog alongside `server.run()` that fires when the captured parent pid changes and `os._exit(0)`s from inside the stdio context manager, bypassing the same wedged teardown ([#2185](https://github.com/headroomlabs-ai/headroom/issues/2185), [#1761](https://github.com/headroomlabs-ai/headroom/issues/1761)).
- **cache/prefix-freeze:** resolve `PrefixCacheTracker`s per conversation lineage within a session id, so concurrent conversations sharing a fallback id no longer thrash one tracker's frozen-prefix state ([#2085](https://github.com/headroomlabs-ai/headroom/issues/2085)). Without an `x-headroom-session-id` header the fallback id hashes `model + system prompt` — identical across a Claude Code session and every one of its parallel subagents (and any set of sessions reusing one system prompt). On the shared tracker their interleaved histories cross-contaminate the freeze state: the forwarded prefix is byte-unstable on nearly every turn and the provider prompt cache is re-written instead of read — reported as ~4.4x cache-creation inflation and a 2.5–3x net cost increase under Claude Code. `SessionTrackerStore.resolve_tracker` now reuses the tracker whose previous request messages are a prefix of the incoming history (client histories are append-only, so a conversation's next request always extends its previous one), starts a fresh lineage when the history diverges or was rewritten (client-side compaction — that provider cache line is gone anyway), and caps lineages per session id (`PrefixFreezeConfig.max_lineages_per_session`, default 32; over-cap conversations share one overflow tracker instead of evicting established lineages, so a fan-out storm past the cap degrades only its own tail — and `0` disables lineage splitting). Matching compares the original client bytes under the same canonical cross-turn equivalence as the cache-stable delta path (`_canonicalize_for_prefix_compare`), so a moved cache breakpoint, string<->block content sugar, or per-turn transport annotations do not read as a rewrite. Separately, the fallback id now hashes only the LEADING run of `role:"system"` messages: agentic clients interleave `<system-reminder>` turns into the history as actual system-role messages (hook output, skills lists, truncation notices), and hashing those rotated the session id mid-conversation — orphaning the prefix tracker and every other session-sticky subsystem (beta headers, CCR/memory registries, the compression cache) each time a reminder landed. Both handler paths now derive the session id and the lineage from the same original client bytes, so a turn-dependent hook rewrite cannot rotate one without the other. The session id itself never changes: session-sticky state keyed on it (beta-header stickiness, CCR and memory-tool registries, the compression cache) is untouched, and a single-conversation session keeps its exact previous behavior (the first lineage lives under the bare id).
- **proxy/bedrock:** wire `PrefixCacheTracker` updates into both Bedrock backend paths (`handle_anthropic_messages`'s non-streaming branch in `anthropic.py`, and `_stream_response_bedrock` in `streaming.py`). `update_from_response()` was previously only called from the direct-Anthropic-API branch; both Bedrock branches returned before ever reaching it, so the tracker's state stayed permanently empty for the life of a session on any `--backend bedrock` deployment: `extract_cache_stable_delta()` always saw no previous turn, and `--mode cache` fell back to full unmodified passthrough on every turn instead of freezing the already-cached prefix and compressing only the new suffix.
- **install:** `install_supervisor`'s macOS branch did an unconditional `launchctl bootout` followed by a bare `bootstrap` with no retry, unlike `start_supervisor`, which already rides out the ~15s EIO (error 5) window launchd exhibits for several seconds after a bootout. This left `install apply`'s own reinstall path (and anything that re-applies a deployment, e.g. a future `headroom doctor --fix`) exposed to a race that previously required manual recovery (bootout + remove the plist + reapply). Extracted the retry loop already used by `start_supervisor` into a shared `_bootstrap_with_retry()` helper, now used by both call sites.
diff --git a/headroom/ccr/mcp_server.py b/headroom/ccr/mcp_server.py
index f6f771af6..9fa2b69ad 100644
--- a/headroom/ccr/mcp_server.py
+++ b/headroom/ccr/mcp_server.py
@@ -24,6 +24,7 @@ from __future__ import annotations
import argparse
import asyncio
+import contextlib
import json
import logging
import os
@@ -87,6 +88,12 @@ _READ_ENABLED = os.environ.get("HEADROOM_MCP_READ", "off").lower().strip() in (
DEFAULT_PROXY_URL = os.environ.get("HEADROOM_PROXY_URL", "http://127.0.0.1:8787")
+# How often the parent-death watchdog polls os.getppid() (seconds). When the
+# launching MCP client is SIGKILLed, stdin EOF may never arrive and the SDK's
+# blocking stdin-reader thread wedges server.run() forever, orphaning this
+# process under init/launchd. The watchdog reaps us once we are reparented.
+PARENT_DEATH_POLL_INTERVAL = 5.0
+
def _format_session_summary(
summary: dict[str, Any],
@@ -1036,16 +1043,71 @@ class HeadroomMCPServer:
)
]
- async def run_stdio(self) -> None:
- """Run the server with stdio transport."""
+ async def _await_parent_death(self, interval: float) -> None:
+ """Resolve once the launching parent process is gone.
+
+ We are started with a real parent (the MCP client, over stdio). If our
+ ppid changes we have been reparented — the client died — and stdin EOF
+ may never come (SIGKILL leaves the SDK's blocking stdin reader wedged),
+ so ``server.run`` would hang forever. Detecting the reparent lets the
+ caller shut down instead of orphaning. Watching for *any* change to the
+ captured ppid is more portable than hard-coding ``== 1``: POSIX reparents
+ to init/launchd (pid 1), but a Linux subreaper can adopt us instead.
+ """
+ initial_ppid = os.getppid()
+ while os.getppid() == initial_ppid:
+ await asyncio.sleep(interval)
+ logger.warning(
+ "parent process gone (ppid %s -> %s); shutting down MCP server",
+ initial_ppid,
+ os.getppid(),
+ )
+
+ async def run_stdio(
+ self,
+ parent_death_poll_interval: float = PARENT_DEATH_POLL_INTERVAL,
+ ) -> None:
+ """Run the server with stdio transport.
+
+ Normally the server stops on stdin EOF when the client disconnects. A
+ parent-death watchdog runs alongside it because an abrupt client SIGKILL
+ leaves the SDK's stdin reader wedged and ``server.run`` never returns; the
+ watchdog forces shutdown once we are reparented.
+ """
async with stdio_server() as (read_stream, write_stream):
logger.info(f"Headroom MCP Server starting (proxy: {self.proxy_url})")
- await self.server.run(
- read_stream,
- write_stream,
- self.server.create_initialization_options(),
+ serve_task = asyncio.create_task(
+ self.server.run(
+ read_stream,
+ write_stream,
+ self.server.create_initialization_options(),
+ )
+ )
+ watchdog = asyncio.create_task(self._await_parent_death(parent_death_poll_interval))
+ done, _pending = await asyncio.wait(
+ {serve_task, watchdog},
+ return_when=asyncio.FIRST_COMPLETED,
)
+ if watchdog in done:
+ # Parent died. stdin EOF may never arrive — a client SIGKILL
+ # leaves the SDK's stdin-reader thread blocked, so both
+ # ``server.run`` and the ``stdio_server`` context-manager exit
+ # can hang forever. Reap from *inside* the context manager:
+ # os._exit skips the (wedged) async teardown and guarantees this
+ # orphan dies. Reached only once the parent is already gone.
+ watchdog.result() # surface a watchdog error, if any
+ await self.cleanup()
+ os._exit(0)
+
+ # Normal shutdown: the client closed stdin and ``server.run``
+ # returned. Cancel the watchdog and let the context manager unwind
+ # cleanly — the reader is not wedged in this path.
+ watchdog.cancel()
+ with contextlib.suppress(asyncio.CancelledError):
+ await watchdog
+ serve_task.result() # re-raise a real server error; no-op on clean EOF
+
async def run_streamable_http(
self,
host: str,
diff --git a/tests/test_ccr_mcp_server.py b/tests/test_ccr_mcp_server.py
index cb8c3a19d..cf9fa1afd 100644
--- a/tests/test_ccr_mcp_server.py
+++ b/tests/test_ccr_mcp_server.py
@@ -1,6 +1,7 @@
from __future__ import annotations
import asyncio
+import contextlib
import json
import pytest
@@ -408,3 +409,86 @@ def test_handle_stats_shows_zero_lifetime_totals_when_present() -> None:
assert "Lifetime Savings:" in text
assert "Tokens saved: 0" in text
assert "Compression savings: $0.00" in text
+
+
+# --- Parent-death watchdog: reap orphaned `mcp serve` on client death --------
+# When the launching MCP client is SIGKILLed, stdin EOF may never arrive and the
+# SDK's blocking stdin reader wedges server.run() forever, orphaning this process
+# under init/launchd. run_stdio() runs a watchdog that detects the reparent and
+# forces shutdown. Refs headroomlabs-ai/headroom#2185 (secondary), #1761.
+
+
+def test_parent_death_watchdog_fires_when_reparented(monkeypatch) -> None:
+ """When ppid changes (client died), the watchdog resolves promptly."""
+ server = mcp_server.HeadroomMCPServer(check_proxy=False)
+ calls = {"n": 0}
+
+ def fake_getppid() -> int:
+ calls["n"] += 1
+ return 500 if calls["n"] == 1 else 1 # captured live, then reparented
+
+ monkeypatch.setattr(mcp_server.os, "getppid", fake_getppid)
+
+ async def run() -> None:
+ await asyncio.wait_for(server._await_parent_death(0.001), timeout=1.0)
+
+ asyncio.run(run()) # returns => detected reparent; TimeoutError would fail
+
+
+def test_parent_death_watchdog_stays_quiet_with_live_parent(monkeypatch) -> None:
+ """A stable ppid must never trip the watchdog."""
+ server = mcp_server.HeadroomMCPServer(check_proxy=False)
+ monkeypatch.setattr(mcp_server.os, "getppid", lambda: 500)
+
+ async def run() -> None:
+ with pytest.raises(asyncio.TimeoutError):
+ await asyncio.wait_for(server._await_parent_death(0.001), timeout=0.05)
+
+ asyncio.run(run())
+
+
+def test_run_stdio_reaps_process_on_parent_death(monkeypatch) -> None:
+ """On reparent, run_stdio cleans up and calls os._exit(0) even though the
+ (stubbed) server.run never returns — the orphan-reaper path."""
+ server = mcp_server.HeadroomMCPServer(check_proxy=False)
+
+ @contextlib.asynccontextmanager
+ async def fake_stdio_server():
+ yield (object(), object())
+
+ monkeypatch.setattr(mcp_server, "stdio_server", fake_stdio_server)
+
+ async def never_returns(*_args, **_kwargs) -> None:
+ await asyncio.sleep(3600) # emulate the wedged SDK reader
+
+ # DummyServer (MCP SDK stub) has no `.run`; raising=False lets us add it.
+ monkeypatch.setattr(server.server, "run", never_returns, raising=False)
+
+ calls = {"n": 0}
+
+ def fake_getppid() -> int:
+ calls["n"] += 1
+ return 500 if calls["n"] == 1 else 1
+
+ monkeypatch.setattr(mcp_server.os, "getppid", fake_getppid)
+
+ cleaned = {"done": False}
+
+ async def fake_cleanup() -> None:
+ cleaned["done"] = True
+
+ monkeypatch.setattr(server, "cleanup", fake_cleanup)
+
+ class _Exited(Exception):
+ pass
+
+ def fake_exit(code: int) -> None:
+ raise _Exited(code) # intercept so pytest survives
+
+ monkeypatch.setattr(mcp_server.os, "_exit", fake_exit)
+
+ with pytest.raises(_Exited) as excinfo:
+ asyncio.run(server.run_stdio(parent_death_poll_interval=0.001))
+
+ assert excinfo.value.args[0] == 0
+ assert cleaned["done"] is True