Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Model Hub inference deadlines
Problem and contract
The September 8, 2026 investigation found that Model Hub reused its 60-second transport budget for response headers, the first response bytes, first model output, and completion of buffered inference. A connected, healthy provider can take longer at any of these stages. Retrying discards its work and can exhaust the caller's retries without ever allowing the model to finish.
An admitted inference request must remain live until protocol completion, a concrete transport failure, cancellation by its owner, or, for the streams the stream idle bound covers, silence past that bound. Elapsed model time alone is not failure. This property applies to every supported protocol and to both streaming and buffered responses. Cancellation must still release the connection, temporary storage, and source ownership, and a waiting request must not block unrelated requests.
Scope
- Remove inference wall-clock deadlines in the engine client.
- Retain bounded connection establishment, management/discovery calls, reading an already-failed HTTP response, and resource cleanup.
- Start the finite local buffered-response parsing budget after the complete response has arrived, so model execution cannot consume it.
- Audit backend idle reclamation, gateway cancellation, and native caller timeouts before closing the change. Record evidence and scope decisions here.
- Treat temporary Source failures and cooldowns as delayed admission under one service-owned 120-second recovery window per pending model request. Re-read configuration after cancellable waits and share the window across fallback passes. Retry pre-output failures only; never replay model output.
- Treat cooldown expiry as eligibility, not success. One pending real request owns half-open admission; only recognized output or verified protocol success records recovery. Window expiry never cancels admitted inference.
- Pass all-temporary Hub supply through native preflight immediately, so native acceptance and the first model request do not own independent wait windows.
- Recognize Responses reasoning-text and compatible Chat reasoning-content deltas as model output using the existing protocol taxonomy.
- Keep production configuration and the running service unchanged during development; verify with hermetic tests and local Incus where applicable.
Validation
Exercise delayed inference at each wire boundary beyond the connection budget, then assert successful completion or owner-driven cancellation. Preserve error classification, usage, response replay, source isolation, and shutdown tests.
Audit inventory
| Boundary | Initial evidence | Decision |
|---|---|---|
| Engine response headers / first bytes / first output | Uses transport timeout | Remove inference deadline |
| Buffered inference completion | Uses transport timeout | Remove inference deadline |
| Buffered local parsing | Shares deadline with body download | Give local parsing its own budget |
| Gateway teardown and cancellation drain | Runs after completion/cancellation | Retain cleanup bounds |
| Codex and Claude active-session reclamation | Age backstop overrides durable active ownership | Require the existing ownership veto at both reclamation checks; retain repair of orphaned local flags |
| Codex SSE idle | Native default is 300,000 ms, after HTTP headers | Retain the native policy; zero expires immediately, not disabled. No arbitrary huge override and no early gateway headers |
| Source cooldown | A waiting route returns 409; Codex does not honor HTTP Retry-After | One shared recovery window covers pre-output retries and delayed admission; terminal compatibility maps only recovery exhaustion to Codex 400 or Claude/OpenCode 424, with no Retry-After |
| Prelude output classification | Responses reasoning text and Chat reasoning content were treated as metadata | Extend the existing taxonomy and acceptance fixtures |
| Prelude storage | Memory spills at 256 KiB; replay has deliberately no total byte ceiling | Preserve lossless replay and cancellation cleanup, not a new response-size rejection policy |
| Codex local control RPC | 120-second acknowledgement deadline; turns run via notifications | Retain, not an inference deadline; resetting initialization on one slow RPC is a separate transport issue |
| OpenCode active turn | Default duration cap already disabled | No change |
| OpenCode admission and stale overlay cleanup | 120-second acceptance check and 30-second old-process drain | Retain; neither limits an accepted live turn |
| HTTP request body | 16 MiB input cap; oversized requests use aiohttp's 413 | Retain size protection; protocol-shaped 413 rendering is a separate issue |
Base inspected: 6bad0ca137ca9a72678511a094d366aa7e48af36.
Ownership decision
The ownership snapshot proves an owner still exists, not that its process is healthy. The old age backstop has no discriminator between a silent valid turn and a live-but-deadlocked backend. Narrowing the veto to ignore the resource's own active Turn would therefore restore the same demonstrated false eviction. This change deliberately prioritizes owner-controlled termination. Explicit Stop, native terminal/EOF handling, and reclamation of unowned local flags remain. A live deadlock with no terminal evidence can remain until explicit Stop; do not claim that this change introduces automatic deadlock detection. Such detection needs independent backend evidence, not another elapsed-inference threshold.
Incident evidence
Local service logs and durable Turn rows were compared for the two reported Sessions. Times below are UTC+08:00 on September 8, 2026.
sesssbbj4mt2y: forced progress-timeout settlement at 00:22:37, 01:21:07, 09:00:12, and 09:50:27, with native reconnect messages before settlement.sesvmgbdub2gp: the same forced settlement at 01:31:15. A late native 503 followed at 01:34:35; both durable rows name the same native Turn.sesssbbj4mt2ylikewise recorded a late 503 at 09:57:16 against the native Turn forcibly settled at 09:50:27. These rows are not independent upstream outages and must not be counted as such.- Another observed call cooled its sole source at 09:14:52 until 09:15:22, while the caller exhausted reconnects by 09:14:56 and failed with 409.
- The generic 409 at 09:58:14 in
sesvmgbdub2gphas no attributable gateway provenance. It must not be asserted to be the cooldown case without evidence. Route/configuration conflicts remain terminal; their safeguards are unchanged.
Native caller evidence
Verified against the installed Codex 0.153.2 and its exact official source tag:
- HTTP retry implementation uses local backoff and does not read Retry-After.
- Native regression test explicitly asserts local backoff despite a Retry-After header.
- SSE consumer applies its idle deadline to the next parsed SSE event, after headers. SSE comment heartbeats do not necessarily reset that event-level deadline. The gateway still defers headers until model output or a protocol terminal, preserving pre-output fallback and HTTP error semantics.
The native SSE cap and a provider's own limits remain outside the removed Avibe inference deadlines; this PR does not claim end-to-end infinite patience.
Bounded recovery compatibility amendment — September 9, 2026
The approved binary audit in model-hub-retry-experience.md supersedes the
earlier once-only gateway cooldown wait. Codex 0.153.2 issued six requests for
424/422 despite provider retries being zero and unbounded connection retries
being disabled; 400 produced one. Claude 2.1.263 and OpenCode 1.18.18 produced
one for 424. All three accepted headers delayed by 121 seconds. These are
orchestrator-owned binary audit results, not a claim that core unit tests ran
the native binaries.
The single exhausted domain type/code is model_hub_recovery_exhausted with
neutral wire text, mapped by backend (not upstream protocol). Codex's internal
400/InvalidRequest transport interpretation is intentional compatibility, not
the public invalid_request_error classification. Original HTTP status and
Source failure reasons remain in provenance. Only exhausted automatic recovery
uses this branch; auth, request incompatibility, engine errors and post-output
failure retain their established behavior.
Hub-only launch injection sets Codex provider request_max_retries=0 and
features.unbounded_connection_retries=false, plus Claude environment and
settings CLAUDE_CODE_MAX_RETRIES=0. Codex stream_max_retries, Claude watchdog,
and normal continuation remain unchanged. The OpenCode native poll-loop
consumer and all native terminal/rendering integration are orchestrator-owned.
The received-terminal audit does not prove exactly-once behavior if the terminal response is lost. Existing request identity does not identify every native invocation across arrivals. No prompt-based dedupe, Turn blackout or Source blackout is introduced to hide that transport residual.
Validation evidence
- Real loopback HTTP tests cover all three protocols, five delayed inference boundaries, and completion/cancellation. They assert no timeout or cooldown classification for slow successful inference and closed client resources.
- Gateway HTTP tests cover known cooldown, repeated failure, configuration changes during the wait, and cancellation for every backend.
- Existing protocol classification, byte replay, usage, teardown, native termination, and stale-local-flag repair suites remain required.
- Focused combined local validation: 1,223 passed, 30 subtests passed, one previously expected xfail. Changed Python files pass Ruff.
- Production configuration, credentials, and installed service are unchanged.
Review inventory and scope decisions
The orchestrator fetched every review thread, review verdict, and matching lint
run before this revision. Review 5136689855 examined
2d0700c64c849ba97e71ec7b4cbc7f9cf9522492 and raised three findings:
| Root-cause class | Finding | Complete correction |
|---|---|---|
| Timestamp compatibility | Valid UTC Z strings fail on Python 3.10 |
Normalize the offset before parsing; test both accepted UTC forms through source resolution and the gateway |
| Cancellation ownership | Transport admission shields an unbounded upstream wait | Propagate cancellation until a handle is acquired; retain shielding for finite settlement; carry already-observed wire facts on cancellation for usage accounting |
| Retry provenance | A previous no-candidate marker wins over a later admitted attempt | Clear obsolete supply facts and their terminal projection at admission; cover later success, failure, and cancellation |
This is the first findings-bearing head for all three classes; no repeated-class
circuit breaker has tripped. No architecture or persistent data-model rewrite is
required. The adapter's cancellation exception extends CancelledError with
already-observed wire facts only; it introduces no source failure, retry, or new
terminal outcome. Its canonical interface and mirror change together with both
consumers. Unbounded inference remains cancellable; finite ledger/resource work
retains its existing owner and limits.
The expanded local regression passed 1,410 tests and 30 subtests, with one pre-existing xfail; changed Python files pass Ruff and whitespace checks.
The new complete gateway-to-service-to-adapter-to-HTTP test reproduced the original leak: downstream cancellation returned while the loopback provider request remained open. It now verifies every wire wait and protocol, transport lease release, and exactly-once accounting of pre-output usage. Existing tests for a response beating cancellation now synchronize on actual handle availability, not mere transport admission.
Local Incus installation, startup, and health verification completed on the first
head, with 803 passing tests, 30 passing subtests, and one existing xfail. Only
the task-specific avr-wt-model-hub-inference-deadlines environment was used.
The first lint run's migration-fixture failure is supplied by merged dependency
#1933 (a58644528deb8365876915904d11fb89df48db11); integrate it normally before
requesting the next exact-head review and CI.
Second-head circuit breaker and orchestrator decision
The full paginated inventory contains five findings across two reviewed heads:
2d0700c64c849ba97e71ec7b4cbc7f9cf9522492 (review 5136689855) and
bb0ba11f056084884231c89814084b23c07594ad (review 5136913971). The first
head's three findings are listed above. The second head repeats two classes:
First-head threads: PRRT_kwDOPbFPYs6gEzJI (timestamp),
PRRT_kwDOPbFPYs6gEzJM (cancellation), and PRRT_kwDOPbFPYs6gEzJN (provenance).
| Root-cause class | Second-head finding | Full-chain diagnosis |
|---|---|---|
| Timestamp semantics | Thread PRRT_kwDOPbFPYs6gFJW0: string ordering selects the wrong cooldown deadline |
Validation preserves offsets. Parsing at the gateway alone does not fix selection in turn_supply_facts; the probe API has the same string minimum. Resolver readiness, service comparisons, and gateway delay must share timestamp semantics |
| Retry provenance | Thread PRRT_kwDOPbFPYs6gFJW2: an earlier exhausted projection overrides Stop after readmission |
Admission clears only no-candidate projections and only when a supply marker exists. Exhaustion has no such marker. All supply-level retry projections need the same admission boundary, independently of that marker |
Both repeated classes reached the two-head threshold. The orchestrator paused editing, inspected all five threads and their consuming paths/tests, and chose the following smallest complete correction before resuming:
- Move the existing service timestamp parser to the existing resolver module.
Reuse it for readiness, source/OAuth time comparisons, both earliest-deadline
selections, and gateway delay. Compare aware instants, preserving the selected
original string for rendering and compatibility. Keep Python 3.10
Zsupport and the existing interpretation of old naive timestamps as UTC. - On actual attempt admission, retire both
no_candidateandexhaustedprojections independently of supply-marker presence. Preserve failed-attempt history, request-specific identities, committed served/nonretryable facts, and the Stop/frozen-outcome boundary. No persistent schema or lifecycle rewrite. - Enumerate every supply-level outcome/variant from the production authority in the retry regression, followed by success, exhaustion, terminal failure, or Stop. Exercise exhaustion, delayed recovery, readmission, and Stop through real gateway HTTP as well. Exercise mixed offsets, fractional timestamps, and route permutations through resolution, probe, and gateway admission.
- Keep the resolver-consuming retry test in the existing L3 ownership scope.
Current-head lint run
34182012889failed only because its prior placement intest_model_hub_provenance.pyadded an unregistered resolver importer (O1). Move the test to L3; do not weaken the authority guard or expand lane ownership.
The first lint run 34179699562 belongs to the old head and its dependency
failure is resolved by the already-integrated #1933. The current head has 15
successful checks and the O1 failure plus its failing unit-test aggregate.
Cancellation ownership did not recur on the second reviewed head. The durable
watch and cursor remain unchanged. Production, merge, and deployment stay out
of scope; fresh exact-head review, complete lint, and task-local Incus validation
are still required after this correction.
Before the correction, the added consuming tests reproduced 12 failures across chronological selection, old naive timestamps, and exhaustion followed by Stop (including Codex and Claude HTTP requests). Afterward all 75 focused cases pass. The combined local regression passes 1,492 tests and 30 subtests with one existing xfail, including the previously failing config/authority closure test. The authority checker reports no findings; changed Python files pass Ruff and whitespace checks. The orchestrator inspected the parser/minimum diff and its probe-to-gateway consumer, and the admission diff and its HTTP Stop consumer.
Local projection budget follow-up (2026-09-27)
The audit decision "give local parsing its own budget" was only half applied:
the buffered success path started its parse deadline after the body arrived but
sized it with the client's connection budget, and the failed-response path
shared one deadline between the error-body read and its projection. The
inference-wait regression uses a 20 ms connection budget, so any scheduling or
GC pause of that length while parsing an already-received body became a
TIMEOUT, a 1-second source cooldown, and a gateway 503. That is the recurring
gateway-complete-buffered_body-openai_responses CI failure on master.
Both projections now start their own _LOCAL_PROJECTION_TIMEOUT_SECONDS budget
after the body has arrived. The connection bound and the bounded read of an
already-failed response are unchanged, and production timing is unchanged
because that budget equals the 60-second transport default. A regression drives
a projection that outlasts the connection budget and asserts success; the
projection-budget test asserts the budget is independent of the client timeout.
Stream idle bound (2026-10-13)
Problem
The engine sets no timeout once an upstream connection is established, not even an opt-in HTTP/2 health check. Its maintainers closed router-for-me/CLIProxyAPI#6534 as not planned: timeouts and cancellation belong to the downstream client. For the engine that client is this gateway.
Engine patches 4 and 5 (patches/model-hub-cpa-v8.0.30/README.md) are carried
by design and turn a dead network path into an error in about 45 s. They cannot
see a relay that stays alive while its own upstream is stuck: the relay answers
PINGs and keeps writing : keep-alive comments, and the engine holds
comment-only frames until the next event. Such a stream sent the gateway nothing
until the caller gave up: Vibey after 600 s, Codex after 300 s between events,
and Claude Code after its own byte-silence watchdog. The rc3 turn
trn_98d57e9e6ea0 waited the full 600 s on the API-key relay; whether its path
was dead or its upstream stuck was not recorded.
Contract
- A streamed request to a Source in the table below ends when the engine sends the gateway no byte for 120 s. Every byte restarts the bound: data events, pings and comments alike. Native callers apply it from the moment the request body is written, so it covers the wait for response headers too.
- When the bound runs out, the gateway closes the engine connection. The engine cancels its upstream request through its request context; measured on the shipped v8.0.23-3 engine, the upstream saw the disconnect within 0.02 s.
- The attempt ends as a
timeoutoutcome, classified as thenetworkfallback reason: Source recovery delays of 1, 2, 4 ... s, and no persisted health write.- Before model output the caller has seen nothing. Model Hub retries
inside its 120 s recovery window, which opens at this failure, not at the
request. The caller receives the retried stream, or recovery exhaustion
with cause
network. - After model output nothing is replayed. The stream ends with the
protocol's
stream_interruptedterminal, as for any transport failure after output, and the caller's own policy applies.
- Before model output the caller has seen nothing. Model Hub retries
inside its 120 s recovery window, which opens at this failure, not at the
request. The caller receives the retried stream, or recovery exhaustion
with cause
- Buffered responses and every Source outside the table keep the original contract: elapsed time and silence alone are not failure.
| Source vendor | Protocol, untranslated | Bounded | Longest healthy silence measured |
|---|---|---|---|
anthropic |
anthropic |
yes | 10.0 s: a ping every 10 s through 336 s of buffered thinking, a 157K-token prompt, and a 12K-token tool input that the relay buffered |
openai |
openai_responses |
yes | 35.8 s, before the first event of a cold 102K-token prompt; at most 29.2 s between events at max effort, with or without a reasoning summary; tool-call arguments streamed as deltas |
xai |
openai_responses |
no | 188 s: grok-4.6 sent a 15K-token tool call in one delta, and the engine held the relay's comments meanwhile |
| any | translated by the engine | no | 72.8 s (Responses to Claude) and 141.8 s (Chat to Claude): the translators drop the Anthropic pings |
| any other | openai_chat, google, or another vendor |
no | not measured |
Measured on 2026-10-13 through the local engine (v8.0.23-3, darwin-arm64) with
long-reasoning prompts at each model's highest accepted effort. The streams
were 33 to 900 s long. Only timing metadata was kept, outside this repository.
The ChatGPT subscription was capped on that day. Its streams,
measured on 2026-10-12 directly over HTTP/2, had at most 21 s between events,
which agrees with the openai row.
Why the gateway's engine read
- Every Agent's Hub traffic (Claude Code, Codex, OpenCode, Vibey) reaches the
upstream through
EngineClient.invoke. The bound lives there, on the engine connection's read (sock_read), so all of them inherit it. - Before model output the gateway holds the response, so only the gateway sees the upstream's bytes, and only the gateway can retry without the caller seeing a failure. Vibey's own bound sees nothing until the gateway releases headers, and it covers only Vibey.
Interactions
- Model Hub recovery. The 120 s window opens at the first failure. A request whose upstream stays stuck therefore makes about two attempts and ends in recovery exhaustion at roughly 241 s. That is before Vibey's 600 s bound, and for Claude Code shortly after the 240 s early commit, which then carries it in-stream.
- Vibey's retry budget (
max_elapsed_s = 120, measured from the request's first attempt). A stall before output never reaches Vibey's loop. A stall after thinking alone reaches it asinterruptedwithafter_thinking, and is retried once only if 120 s have not passed since the first attempt started. With the bound at 120 s that does not happen, because the error comes at least 120 s after the last byte. After text or a tool call the error is terminal (C-2). It now comes 120 s after the last byte instead of 600 s. - Vibey's 600 s silence bound covers each chunk after the response headers. The wait for the headers belongs to the engine read; see Vibey's wait before output.
- Native callers. Their own bounds (Codex 300 s between events, Claude Code's first-byte and idle watchdogs) are longer and remain unchanged.
- Engine patches 4 and 5 stay. They bound a dead path in about 45 s on every path, the unbounded and translated ones included, and they do so without depending on any upstream cadence.
Known-by-design residuals
- A stuck upstream behind a relay that keeps writing data frames (Anthropic
ping, or anevent: keepaliveframe) resets the bound and is not caught. For the Anthropic relay measured here, the pings may come from the relay itself. - Unbounded Sources keep the caller's bound and patches 4 and 5. Where a healthy stream sends no data for minutes, no byte bound can tell a busy upstream from a stuck one. Forwarding the relay's comments would not help either: a stuck upstream behind a live relay sends those comments too. Holding them back is what lets this bound see such a stall on the bounded Sources.
- The vendor is the Source's declared vendor. A relay declared
openaithat serves a model which buffers tool calls, as xAI does, would be cut at 120 s on a tool call longer than that.
Vibey's wait before output (2026-10-18)
Problem
On 2026-10-17 a Vibey turn on Claude Opus 5.5 at xhigh effort, asked for a
long piece of writing through an anthropic API-key relay, failed after
20 minutes as stalled: two engine requests, each cut by Vibey at 600 s. The
gateway's 120 s bound never fired, so the engine kept receiving bytes, the
Anthropic pings. From Claude Opus 4.7 on, thinking.display defaults to
omitted, so a thinking phase streams no thinking deltas. The gateway withholds
the response headers and every byte until the first content block, and Vibey's
own bound also covered the wait for those headers, so it counted the model's
whole thinking time as silence. Claude Code is not affected: its streams commit
early and carry keepalives (_EARLY_STREAM_COMMIT_SECONDS).
Two alternatives were rejected. Requesting display: "summarized" would make
thinking count as output, but would move the upstream request away from Claude
Code's default shape (owner decision, 2026-10-18). Committing Vibey's streams
early, as Claude Code's are, would drop the x-avibe-served-hop header Vibey
reads with the headers to verify the origin of replayed signatures, and the
HTTP status its error handling keys on.
Contract
- Through the gateway, Vibey does not bound the wait for response headers. Once they arrive, its 600 s bound covers each next chunk as before.
- The engine read holds the bound for every attempt of a Vibey stream: the 120 s idle bound on the measured Sources above, and Vibey's 600 s silence bound on every other stream. Every byte restarts either one, so a model that thinks behind a pinging upstream is never cut, and a silent stream on an unmeasured Source is cut at 600 s and recovered like any other timeout.
- A failure that reaches Vibey before any model output, at least 600 s after its request was sent, ends the turn: it is neither replayed nor continued. Sent again, the request would most likely wait as long again, and Model Hub has already retried what its recovery window allows.
Residuals
- A stuck upstream behind a relay that keeps pinging holds a Vibey turn until Stop, as it holds a Claude Code turn. Elapsed time alone is not failure.
- On an unmeasured Source, after output, the engine read's 600 s bound and
Vibey's own coincide, and the engine read usually ends first, so such a stall
reaches Vibey as
interruptedrather thanstalled. - A persistently silent unmeasured Source still costs about two 600 s attempts (Model Hub recovery), as before (Vibey's own stall and one continuation).
Remote relay request-body prelude (2026-10-31)
The 2026-10-31 relay investigation found Anthropic requests that spent nearly 120 s in the relay's request-body read before the caller cancelled the connection. For Vibey, the engine now uses its existing 600 s caller-owned budget for the request/response prelude when it exceeds the measured 120 s bound. Once the first response byte arrives, measured Anthropic streams retain the 120 s per-read idle bound. Native callers retain the 120 s prelude bound.
This is a client-side resilience margin, not proof that a relay is healthy. The relay still needs its own request-body idle timeout, phase diagnostics, and attempt-ID propagation; those changes are tracked in the relay project.