Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Python agentagent-osaiai-agentsai-partnerchatopsclaudeclaude-codecodexdevtoolsdiscord-botlark-botllmlocal-firstlong-term-memoryopencodeslack-botvibe-codingvibeywechat
fix(model-hub): bound a silent engine stream at the gateway and retry it before output (#2538)
Call Whitefb454ae82e4a
diff --git a/core/agent_core/ai/_common.py b/core/agent_core/ai/_common.pyindex 310dd742c..fa397c9b8 100644--- a/core/agent_core/ai/_common.py+++ b/core/agent_core/ai/_common.py@@ -75,7 +75,10 @@ class WireField:# OpenCode allow 300s of silence; Avibe allows 600s (owner decision,# 2026-10-10), because Model Hub withholds even the response headers until the# first model output, so a high-effort reasoning model's thinking time is all-# silence and can exceed five minutes. Connecting stays separately short, so an+# silence and can exceed five minutes. For the Sources whose streams keep+# sending, the gateway bounds the engine's silence at 120s first and retries a+# stall before output itself (docs/plans/model-hub-inference-deadlines.md), so+# this bound is the outer guard. Connecting stays separately short, so an# unreachable provider still fails fast. Both bounds are shared by every native# adapter; the loop may retry the resulting error only when no model output has# been emitted.diff --git a/docs/plans/model-hub-inference-deadlines.md b/docs/plans/model-hub-inference-deadlines.mdindex 6bc2b6cf6..1694dee03 100644--- a/docs/plans/model-hub-inference-deadlines.md+++ b/docs/plans/model-hub-inference-deadlines.md@@ -9,11 +9,12 @@ take longer at any of these stages. Retrying discards its work and can exhaustthe caller's retries without ever allowing the model to finish.An admitted inference request must remain live until protocol completion, a-concrete transport failure, or cancellation by its owner. Elapsed model time-alone is not failure. This property applies to every supported protocol and to-both streaming and buffered responses. Cancellation must still release the-connection, temporary storage, and source ownership, and a waiting request must-not block unrelated requests.+concrete transport failure, cancellation by its owner, or, for the streams the+[stream idle bound](#stream-idle-bound-2026-10-13) covers, silence past that+bound. Elapsed model time alone is not failure. This property applies to every+supported protocol and to both streaming and buffered responses. Cancellation+must still release the connection, temporary storage, and source ownership, and+a waiting request must not block unrelated requests.## Scope@@ -262,3 +263,105 @@ already-failed response are unchanged, and production timing is unchangedbecause that budget equals the 60-second transport default. A regression drivesa projection that outlasts the connection budget and asserts success; theprojection-budget test asserts the budget is independent of the client timeout.++## Stream idle bound (2026-10-13)++### Problem++The engine sets no timeout once an upstream connection is established, not+even an opt-in HTTP/2 health check. Its maintainers closed+router-for-me/CLIProxyAPI#6534 as not planned: timeouts and cancellation belong+to the downstream client. For the engine that client is this gateway.++Engine patches 4 and 5 (`patches/model-hub-cpa-v8.0.23/README.md`) are carried+by design and turn a dead network path into an error in about 45 s. They cannot+see a relay that stays alive while its own upstream is stuck: the relay answers+PINGs and keeps writing `: keep-alive` comments, and the engine holds+comment-only frames until the next event. Such a stream sent the gateway nothing+until the caller gave up: Vibey after 600 s, Codex after 300 s between events,+and Claude Code after its own byte-silence watchdog. The rc3 turn+`trn_98d57e9e6ea0` waited the full 600 s on the API-key relay; whether its path+was dead or its upstream stuck was not recorded.++### Contract++- A streamed request to a Source in the table below ends when the engine sends+ the gateway no byte for **120 s**. Every byte restarts the bound: data events,+ pings and comments alike. It runs from the moment the request body is written,+ so it covers the wait for response headers too. A paused read, while the+ downstream caller is slower than the engine, does not count.+- When the bound runs out, the gateway closes the engine connection. The engine+ cancels its upstream request through its request context; measured on the+ shipped v8.0.23-3 engine, the upstream saw the disconnect within 0.02 s.+- The attempt ends as a `timeout` outcome, classified as the `network` fallback+ reason: Source recovery delays of 1, 2, 4 ... s, and no persisted health write.+ - **Before model output** the caller has seen nothing. Model Hub retries+ inside its 120 s recovery window, which opens at this failure, not at the+ request. The caller receives the retried stream, or recovery exhaustion+ with cause `network`.+ - **After model output** nothing is replayed. The stream ends with the+ protocol's `stream_interrupted` terminal, as for any transport failure after+ output, and the caller's own policy applies.+- Buffered responses and every Source outside the table keep the original+ contract: elapsed time and silence alone are not failure.++| Source vendor | Protocol, untranslated | Bounded | Longest healthy silence measured |+| --- | --- | --- | --- |+| `anthropic` | `anthropic` | yes | 10.0 s: a `ping` every 10 s through 336 s of buffered thinking, a 157K-token prompt, and a 12K-token tool input that the relay buffered |+| `openai` | `openai_responses` | yes | 35.8 s, before the first event of a cold 102K-token prompt; at most 29.2 s between events at `max` effort, with or without a reasoning summary; tool-call arguments streamed as deltas |+| `xai` | `openai_responses` | no | 188 s: grok-4.6 sent a 15K-token tool call in one delta, and the engine held the relay's comments meanwhile |+| any | translated by the engine | no | 72.8 s (Responses to Claude) and 141.8 s (Chat to Claude): the translators drop the Anthropic pings |+| any other | `openai_chat`, `google`, or another vendor | no | not measured |++Measured on 2026-10-13 through the local engine (v8.0.23-3, darwin-arm64) with+long-reasoning prompts at each model's highest accepted effort. The streams+were 33 to 900 s long. Only timing metadata was kept, outside this repository.+The ChatGPT subscription was capped on that day. Its streams,+measured on 2026-10-12 directly over HTTP/2, had at most 21 s between events,+which agrees with the `openai` row.++### Why the gateway's engine read++- Every Agent's Hub traffic (Claude Code, Codex, OpenCode, Vibey) reaches the+ upstream through `EngineClient.invoke`. The bound lives there, on the engine+ connection's read (`sock_read`), so all of them inherit it.+- Before model output the gateway holds the response, so only the gateway+ sees the upstream's bytes, and only the gateway can retry without the caller+ seeing a failure. Vibey's own bound sees nothing until the gateway releases+ headers, and it covers only Vibey.++### Interactions++- **Model Hub recovery.** The 120 s window opens at the first failure. A request+ whose upstream stays stuck therefore makes about two attempts and ends in+ recovery exhaustion at roughly 241 s. That is before Vibey's 600 s bound, and+ for Claude Code shortly after the 240 s early commit, which then carries it+ in-stream.+- **Vibey's retry budget** (`max_elapsed_s = 120`, measured from the request's+ first attempt). A stall before output never reaches Vibey's loop. A stall+ after thinking alone reaches it as `interrupted` with `after_thinking`, and+ is retried once only if 120 s have not passed since the first attempt+ started. With the bound at 120 s that does not happen, because the error+ comes at least 120 s after the last byte. After text or a tool call the error+ is terminal (C-2). It now comes 120 s after the last byte instead of 600 s.+- **Vibey's 600 s silence bound** stays as the outer guard. It now fires only+ for the unbounded Sources above, or if the gateway itself stops.+- **Native callers.** Their own bounds (Codex 300 s between events, Claude+ Code's first-byte and idle watchdogs) are longer and remain unchanged.+- **Engine patches 4 and 5 stay.** They bound a dead path in about 45 s on+ every path, the unbounded and translated ones included, and they do so+ without depending on any upstream cadence.++### Known-by-design residuals++- A stuck upstream behind a relay that keeps writing data frames (Anthropic+ `ping`, or an `event: keepalive` frame) resets the bound and is not caught. For+ the Anthropic relay measured here, the pings may come from the relay itself.+- Unbounded Sources keep the caller's bound and patches 4 and 5. Where a healthy+ stream sends no data for minutes, no byte bound can tell a busy upstream from+ a stuck one. Forwarding the relay's comments would not help either: a stuck+ upstream behind a live relay sends those comments too. Holding them back is+ what lets this bound see such a stall on the bounded Sources.+- The vendor is the Source's declared vendor. A relay declared `openai` that+ serves a model which buffers tool calls, as xAI does, would be cut at 120 s+ on a tool call longer than that.diff --git a/tests/scenarios/model_hub/catalog.yaml b/tests/scenarios/model_hub/catalog.yamlindex 3fba0f4c0..794b18193 100644--- a/tests/scenarios/model_hub/catalog.yaml+++ b/tests/scenarios/model_hub/catalog.yaml@@ -672,7 +672,7 @@ scenarios:layer: unittest: tests/test_local_deps.py::test_reconcile_startup_dependencies_uses_automatic_runtime_admission- id: MH-RUNTIME-009- name: Connected inference remains live until completion, transport failure, or owner cancellation+ name: Connected inference remains live until completion, transport failure, owner cancellation, or a covered stream's idle boundstatus: coveredkind: timeout_cancellayer: contract@@ -699,6 +699,12 @@ scenarios:test: tests/test_model_hub_retry_lifecycle.py::test_claude_stream_rejected_within_its_first_byte_budget_keeps_the_statusrelated_tests:- tests/test_model_hub_retry_lifecycle.py::test_slow_stream_resolution_keeps_claude_connected+ - id: MH-RUNTIME-013+ name: An engine stream silent past its idle bound is cancelled there, retried before output, and ended in-stream after output+ status: covered+ kind: timeout_cancel+ layer: contract+ test: tests/test_model_hub_retry_lifecycle.py::test_a_silent_engine_stream_is_cancelled_and_retried_at_its_idle_bound- id: MH-USAGE-001name: Tokens an upstream reported before the turn ended are still meteredstatus: covereddiff --git a/tests/scenarios/model_hub/observations.yaml b/tests/scenarios/model_hub/observations.yamlindex 621bb821d..6d11ec702 100644--- a/tests/scenarios/model_hub/observations.yaml+++ b/tests/scenarios/model_hub/observations.yaml@@ -1,6 +1,12 @@version: 2capability: model_hubobservations:+ - id: MH-OBS-014+ source: engine-v8.0.23-3-stream-cadence-and-idle-bound-2026-10-13+ summary: The engine sets no post-connect timeout (CPA #6534 declined), so a relay that stays alive while its upstream is stuck kept a stream silent until the caller's bound. Through the local engine, untranslated Anthropic streams carried a ping every 10 s and OpenAI Responses streams never paused more than 36 s, while xAI sent a 15K-token tool call after 188 s of silence and translated streams dropped the upstream heartbeat for up to 142 s. The gateway now bounds engine silence at 120 s for the measured Anthropic and OpenAI Sources only; on the shipped engine the upstream saw the cancellation within 0.02 s.+ affects:+ - MH-RUNTIME-009+ - MH-RUNTIME-013- id: MH-OBS-013source: claude-cli-2.1.288-thinking-display-relay-rejection-2026-10-03summary: Claude Code 2.1.288 sends thinking.display "updates" with its beta by default, and a relay that rejects it answers HTTP 400. The CLI drops a rejected beta or display value and retries only on an HTTP 400 naming it, but a Hub resolution slower than 2s had already committed a 200 stream, so the rejection arrived as an in-stream error and ended the turn. Claude streams now keep the real status until shortly before the CLI's default first-byte window, and Claude hub launches pin both pre-output timeouts to the CLI maximum so no shorter native choice aborts that wait.diff --git a/tests/test_model_hub_inference_wait.py b/tests/test_model_hub_inference_wait.pyindex 463c56f87..ea16f5964 100644--- a/tests/test_model_hub_inference_wait.py+++ b/tests/test_model_hub_inference_wait.py@@ -65,7 +65,11 @@ def test_connected_inference_waits_for_its_owner(def track_session(**kwargs):timeout = kwargs["timeout"]- assert timeout.total is None and timeout.sock_read is None+ # Elapsed time never ends inference; only a stream's silence does (MH-RUNTIME-013).+ assert timeout.total is None+ # Gateway Sources name their vendor; the bare client fixture is a custom vendor.+ idle_bounded = streaming and entrypoint == "gateway" and protocol in {"anthropic", "openai_responses"}+ assert timeout.sock_read == (client_module._STREAM_IDLE_TIMEOUT_SECONDS if idle_bounded else None)assert timeout.connect == timeout.sock_connect == 0.02session = real_session(**kwargs)sessions.append(session)diff --git a/tests/test_model_hub_retry_lifecycle.py b/tests/test_model_hub_retry_lifecycle.pyindex 8d699ff2f..6db987df3 100644--- a/tests/test_model_hub_retry_lifecycle.py+++ b/tests/test_model_hub_retry_lifecycle.py@@ -19,12 +19,14 @@ from config.v2_config import ModelHubConfig, ModelHubRouteConfig, ModelHubRouteHfrom core.handlers.model_hub.adapter import RawOutcomeKindfrom core.handlers.model_hub.retry import RECOVERY_EXHAUSTED_CODE, RECOVERY_EXHAUSTED_MESSAGEfrom core.handlers.model_hub.service import ModelHubError+from core.handlers.model_hub.stream_wire import ProtocolSSEStatefrom core.handlers.model_hub import turn_gatewayfrom core.handlers.model_hub.turn_gateway import ModelHubTurnGatewayfrom core.run_settlement import SETTLED_BY_TERMINAL_RESULTfrom modules.agents.model_hub import ModelHubRuntimeRouter, bind_launchfrom tests.test_model_hub_l3 import _assert_valid, _canonicalize_fixed_test_routes, _outcome, _sourcefrom tests.test_model_hub_retry_policy import WIRE, clock_service, fail+from vibe.model_hub_runtime import client as engine_client_modulefrom vibe.model_hub_runtime.adapter import CLIProxyEngineAdapterfrom vibe.model_hub_runtime.client import EngineClient, EngineConnectionfrom vibe.model_hub_runtime.state import EngineStateStore@@ -915,3 +917,130 @@ def test_slow_stream_resolution_keeps_claude_connected(monkeypatch, tmp_path, baassert clock.delays == ([] if ending in {"served", "disconnect"} else [30])asyncio.run(run())+++def _large_output(protocol: str) -> bytes:+ """One model-output frame carrying 1 MiB of text in the protocol's shape."""++ text = "x" * (1 << 20)+ if protocol == "anthropic":+ payload = {"type": "content_block_delta", "delta": {"type": "text_delta", "text": text}}+ return b"event: content_block_delta\ndata: " + json.dumps(payload).encode() + b"\n\n"+ payload = {"type": "response.output_text.delta", "delta": text}+ return b"event: response.output_text.delta\ndata: " + json.dumps(payload).encode() + b"\n\n"+++IDLE_BOUNDED_STALLS = ("before_headers", "before_output", "after_output", "heartbeat", "backpressure")+# Large enough to fill every buffer between the engine and a caller that stops reading.+BACKPRESSURE_FRAMES = 24+++ "backend,vendor,source_protocol,stall",+ [+ *(+ (backend, vendor, BACKENDS[backend][1], stall)+ for backend, vendor in (("codex", "openai"), ("claude", "anthropic"))+ for stall in IDLE_BOUNDED_STALLS+ ),+ # Healthy streams that send nothing for minutes: xAI sends a whole tool+ # call at once, the engine's translators drop the upstream heartbeat,+ # and Chat has no measured cadence.+ ("codex", "xai", "openai_responses", "unbounded"),+ ("codex", "anthropic", "anthropic", "unbounded"),+ ("claude", "openai", "openai_responses", "unbounded"),+ ("opencode", "openai", "openai_chat", "unbounded"),+ ],+)+def test_a_silent_engine_stream_is_cancelled_and_retried_at_its_idle_bound(+ monkeypatch, tmp_path, backend, vendor, source_protocol, stall,+):+ """MH-RUNTIME-013: a stream silent past the idle bound ends there, not at the caller's bound.++ The engine sets no timeout once its upstream is connected, so a relay that+ stays alive while its own upstream is stuck sends nothing until the caller+ gives up. The gateway cancels that engine request at its idle bound: before+ model output it retries without the caller seeing anything; after output it+ ends the stream inside the protocol. Any byte, such as an Anthropic ping,+ keeps a slow stream alive, and a caller that stops reading pauses the bound.+ """+ bound = 0.3+ monkeypatch.setattr(engine_client_module, "_STREAM_IDLE_TIMEOUT_SECONDS", bound)++ async def run() -> None:+ _endpoint, protocol = BACKENDS[backend]+ source = _source("src_silent01", "Silent", vendor=vendor, protocol=source_protocol)+ service, clock, models = configured_service(tmp_path, sources=[source])+ metadata, output, terminal, _buffered = WIRE[protocol]+ if stall == "backpressure":+ output = _large_output(protocol) * BACKPRESSURE_FRAMES+ calls: list[dict] = []+ stalled_for: list[float] = []++ async def respond(request: web.Request) -> web.StreamResponse:+ calls.append(await request.json())+ stalls = len(calls) == 1 and stall not in {"heartbeat", "backpressure"}+ started = time.monotonic()+ try:+ if stalls and stall == "before_headers":+ await asyncio.Event().wait()+ response = web.StreamResponse(headers={"Content-Type": "text/event-stream"})+ await response.prepare(request)+ await response.write(metadata)+ if stall == "heartbeat":+ for _ in range(4):+ await asyncio.sleep(bound / 2)+ await response.write(b": keep-alive\n\n")+ if stall == "unbounded":+ await asyncio.sleep(bound * 3)+ if stalls and stall == "before_output":+ started = time.monotonic()+ await asyncio.Event().wait()+ await response.write(output)+ if stalls and stall == "after_output":+ started = time.monotonic()+ await asyncio.Event().wait()+ await response.write(terminal)+ await response.write_eof()+ return response+ except asyncio.CancelledError:+ stalled_for.append(time.monotonic() - started)+ raise++ async with loopback_engine(tmp_path, service, respond) as (adapter, _engine_client):+ async with gateway_client(service, backend, models[backend]) as (_gateway, client, url, headers):++ async def exchange() -> tuple[int, bytes]:+ async with client.post(+ url, headers=headers, json={"model": "shared-model", "stream": True},+ ) as response:+ if stall == "backpressure":+ # The engine keeps sending while the caller reads nothing.+ await asyncio.sleep(bound * 3)+ return response.status, await response.read()++ # Without the idle bound the stalled engine request never ends.+ status, body = await asyncio.wait_for(exchange(), timeout=5)+ assert status == 200+ assert adapter._active_transports == 0++ if stall in {"heartbeat", "unbounded", "backpressure"}:+ keepalives = b": keep-alive\n\n" * 4 if stall == "heartbeat" else b""+ assert body == metadata + keepalives + output + terminal+ assert len(calls) == 1 and stalled_for == []+ return+ # The engine request itself was cancelled, once, at the bound.+ assert len(stalled_for) == 1 and bound * 0.9 <= stalled_for[0] < bound + 2+ if stall == "after_output":+ # Output already reached the caller, so the stream ends in-protocol instead of replaying.+ assert len(calls) == 1+ assert body.startswith(metadata + output)+ ending = ProtocolSSEState(protocol)+ ending.observe(body[len(metadata + output):])+ assert ending.terminal_outcome == "failed_terminal"+ else:+ assert len(calls) == 2+ assert body == metadata + output + terminal+ assert clock.delays == [1]++ asyncio.run(run())diff --git a/vibe/model_hub_runtime/client.py b/vibe/model_hub_runtime/client.pyindex e4e35c6d5..3b896e553 100644--- a/vibe/model_hub_runtime/client.py+++ b/vibe/model_hub_runtime/client.py@@ -79,6 +79,18 @@ _ERROR_OBSERVATION_BYTES = 256 * 1024# Projecting a response body that has already arrived is local work. It has# its own budget so the client's connection budget never limits it._LOCAL_PROJECTION_TIMEOUT_SECONDS = ENGINE_TRANSPORT_TIMEOUT_SECONDS+# A streamed response that sends the gateway no byte for this long is stalled.+# The engine sets no timeout once its upstream is connected, so the gateway,+# its downstream client, bounds it. Only (Source vendor, protocol) pairs whose+# untranslated streams were measured to keep sending in every phase are+# bounded: Anthropic sends a ping every 10 s, even while a relay buffers a tool+# call's input, and OpenAI's Responses events never paused for more than 36 s,+# including before the first event of a cold 250K-token prompt. The engine+# holds SSE comments and its translators drop heartbeats, so an xAI tool call+# or a translated stream can be silent for minutes while healthy+# (docs/plans/model-hub-inference-deadlines.md, "Stream idle bound").+_STREAM_IDLE_TIMEOUT_SECONDS = 120.0+_STREAM_IDLE_BOUNDED_SOURCES = frozenset({("anthropic", "anthropic"), ("openai", "openai_responses")})_OFFICIAL_BASE_URLS = official_api_key_base_urls()logger = logging.getLogger(__name__)_ProjectedJSON = TypeVar("_ProjectedJSON")@@ -479,11 +491,20 @@ class EngineClient:total=None,connect=self.timeout,sock_connect=self.timeout,- sock_read=None,+ sock_read=(+ _STREAM_IDLE_TIMEOUT_SECONDS+ if stream and request_protocol == source.protocol+ and (source.vendor, source.protocol) in _STREAM_IDLE_BOUNDED_SOURCES+ else None+ ),)- # Connecting to the local engine is bounded. Once connected, headers- # and response bytes can wait on upstream inference for any duration;- # completion, transport failure, or owner cancellation ends that wait.+ # Connecting to the local engine is bounded. Once connected, inference+ # can take any duration; completion, transport failure, or owner+ # cancellation ends it. A stream whose upstream keeps sending must keep+ # sending, though: from the request on, every byte restarts its idle+ # bound, and one that runs out closes the engine connection, which+ # cancels the upstream request. A buffered response sends nothing until+ # it is complete, so it has none.trace_configs: list[aiohttp.TraceConfig] = []if on_request_sent is not None:# CPA resolves the route as soon as it reads the body, while a@@ -714,7 +735,8 @@ class EngineClient:if wire_state is not None:raise InvokeCancelledError(wire_state) from cancelledraise- except asyncio.TimeoutError:+ except asyncio.TimeoutError as exc:+ logger.warning("Model Hub engine request for source %s timed out: %s", source.source_id, exc)if response is not None:response.close()await session.close()@@ -1599,7 +1621,7 @@ async def _response_stream(message="upstream stream ended before a protocol terminal event",stream_started=wire_state.model_output_started,)- except asyncio.TimeoutError:+ except asyncio.TimeoutError as exc:outcome = _observed_stream_terminal_outcome(wire_state,source,@@ -1607,6 +1629,9 @@ async def _response_stream(response.status,)if outcome is None:+ logger.warning(+ "Model Hub engine stream for source %s timed out after model output: %s", source.source_id, exc,+ )outcome = _outcome(kind=RawOutcomeKind.TIMEOUT,source=source,