Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
gateway: a translated stream whose provider goes quiet keeps its client alive for up to 30 minutes (youyo.p)
yetone13f658a37da5
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex 6540032c5..4f499b951 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -178,7 +178,7 @@ recorded as a prediction failure (`noPrediction`).- A 2xx that can't be an API's answer is the 502 it stands for (`notAnAPIReply`, #1012): a web page (`text/html`, or a body that begins as one: a sign-in page, Cloudflare's challenge), a body with nothing in it, or one sent as JSON that doesn't begin as JSON. Every upstream request goes through it (`forwardOnce`, and the built-in clients' own: the ChatGPT backend's `/responses`), so it fails over, rests and is logged as a 502 does, its reason naming the page's title.- An error status whose body holds no message of its own says what it was (`provider.Unsaid`, through `APIError` and the plan-quota and drawing readers): a web page by its title as plain words, a firewall's block page (`EdgeBlocked`: Alibaba Cloud's, Cloudflare's "you have been blocked", "used Cloudflare to restrict access" and `error code: 1005`–`1020`) with `BlockedHint` after it, and an empty body under an HTTP status as refused with no details, often a firewall or proxy in front of the API (`NoDetails`; an empty `response.failed` event, whose fallback is no status, still reads "upstream stream failed"). A bare "403 Forbidden" told a relay's user nothing (maralica on Discord). A JSON error still reads as its message. `TestAPIErrorUnsaid`, `TestRelayRefusalSaysWhatItWas`. Only the first bytes are peeked, which a stream waits for anyway, and a compressed body is left as it came.- A 2xx from Zhipu or Z.ai holding the platform's refusal envelope (`{"code":1000,"msg":…,"success":false}`, which their Responses API at `/api/v1` answers a refused key) is the refusal it stands for (`provider.ZhipuEnveloped` in `forwardOnce`): 401 for a key refused, 429 for a limit or no balance, as Zhipu's error codes give, so it fails over and rests as those do and the agent is told why. `TestZhipuResponsesRefusalIsRefusal`.-- A streamed request is never left silent while its vendor says nothing. `holdWriter.watch` runs beside each try: past `keepHeldAfter` (15s) it sends a held try's agent the stream's 200 and SSE comments (`keepAlive`), none of the try, so another candidate may still answer in the same stream; once the reply passes, it puts a comment between two of its lines after `keepaliveEvery` of quiet. It stops after `keepaliveLongest` (5 minutes) with nothing from the vendor. A try waiting for its key's or account's slot or for room in its minute (`lanes.take`, `rpms.wait`) is kept alive the same way (`keepQueued`), for as long as it waits; once the 200 is out, the try is held, and its failure or its queue turning it away reaches the agent as the stream's error (`failTo`), not a status (coeo91 on Discord). A wait shorter than `keepHeldAfter` sends nothing, so its 429 keeps its `Retry-After`. A proxy in front such as Cloudflare (125s) otherwise ends the request (#947). Gemini streams get no comments (#934). A vendor that then ignores `stream:true` and answers 200 with its reply whole has it sent as the agent's stream (`wholeAsStream`: Chat, Responses, Anthropic), not as the stream's error, while a reply under a type that says neither JSON nor a stream is held as a stream when its first bytes are one (`sseStart`), read event by event like one that says so, so an error among its frames still goes to the next candidate (#1418: it went through at once); a 200 that is no reply of the protocol's fails as a translated reply that didn't stream does (502, "… did not stream"), for the next candidate to answer, and the last one's reaches the agent as the stream's error. A held stream that has sent nothing but the frames before a reply (`response.created`, `response.in_progress`, `codex.rate_limits`, pings, an empty part begun, and any event `streamEvent` doesn't know to carry text, reasoning, a call or a reply's end (`replyEvent`), or an item not known that isn't a call or a tool's (`streamPart.said`): `eventLead`; Aliceonly on #1418: an unknown event was taken for content and let a frames-only 200 through, and its `server_is_overloaded` reached Codex with five candidates left) stays held as long as its agent is kept alive, up to `holdLead` (4 minutes, under the 300s Codex waits for an event; `keptQuiet`), so a `server_is_overloaded` that comes after it is a failure the next candidate answers (#1418: let through at 15s, the ChatGPT backend's failure 36s in ended Codex's turn with sub2api, next in the group, never asked). A stream whose agent can't be kept alive (Gemini's) is let through after `holdLongest` (15s), as before. Codex counts only events toward that 300s, not comments, so the bound is the request's, not each try's: once Codex on Responses has heard no event for `agentQuietMost` (4 minutes) since the request began or its last event (`keptAlive.tooQuiet`), the try in hand is let through (`scan`, `keepQuiet`), and no group replan or same-one retry is taken whose pause would end past it (fadenoob on Discord: held tries and replan pauses added up past 300s and Codex reconnected on "idle timeout waiting for SSE"). A translated reply whose upstream stream stops before it said its reply had — an Anthropic one with no stop_reason and no message_stop, a Responses one with no `response.completed`, the same two a relayed one is told by (`sniff.whole`; a Chat stream may end without `[DONE]` and be whole), a Gemini or Code Assist one whose last chunk has no candidate's `finishReason` (`geminiEnds`, #1643: `usageMetadata` is no end, Vertex AI sends it on every chunk; a chunk with no candidate at all is the empty reply's to answer, #667), and a relayed Gemini one told the same way (#1644; it ends in Gemini's own `{"error":{"code":502,…,"status":"UNAVAILABLE"}}` data line from `streamFailure`, `TestGeminiRelayEndedShort`; whatever a relayed stream gets appended, that error or a filled-in `finish_reason`, begins after a blank line, two newlines when the upstream's last write stopped inside a line, so it is an event of its own rather than more of the half-written one, #2446, `TestPassthroughStreamCutMidLine`, `TestGeminiCompatFinishAfterPartialLine`), and a Chat one from Gemini's OpenAI-compatible API (`geminiCompat`) with no `finish_reason`, relayed or translated (`saysItsEnd`; Linka on Discord: the upstream hung up between a whole function call and its `finish_reason`, and Pi threw the call away as "Stream ended without finish_reason"; that API counts the usage only once the generation is done, so when the last chunk carried it only the `finish_reason` was lost and it is filled in, `tool_calls` after a call and `stop` otherwise, as a relayed chunk before `[DONE]` or a translated reply's end, and the reply counts as whole (`chatTail`); only one with no usage at its end is a cut, `TestGeminiCompatRelayEndedShort`, `TestGeminiCompatTranslatedEndedShort`) — ends the turn with "the reply ended before it was complete" (`errEndedShort`, `TestResponsesUpstreamEndedShort`, `TestGeminiUpstreamEndedShort`), asked again in place where the upstream goes on from a prefill (`continuation.go`). A client that asked for no stream is told the same 502 of such a cut (`translate`, `TestNonStreamEndedShort`, `TestGeminiNonStreamEndedShort`), rather than handed what came as its whole reply. A translated reply from an Anthropic or Responses upstream that ended but said nothing (no text, call, search or image) and counted no token at all is no reply either, whatever its end says (`repliesCounted`, #2086: an Anthropic-compatible proxy's 200 ended a Codex turn as `response.completed` with no output, no usage and no served model, and Codex didn't ask again). It is the empty reply's to answer (#667): asked again in place, then failed with "an empty reply", streamed or not, for every client (`TestUncountedEmptyReplyFails`, `TestUncountedEmptyReplyFailsEveryClient`). Claude's own empty `end_turn` counts its prompt and stays a whole reply (`TestWellFormedStreamsStillComplete`). A reply passed through in its own protocol isn't read this way.+- A streamed request is never left silent while its vendor says nothing. `holdWriter.watch` runs beside each try: past `keepHeldAfter` (15s) it sends a held try's agent the stream's 200 and SSE comments (`keepAlive`), none of the try, so another candidate may still answer in the same stream; once the reply passes, it puts a comment between two of its lines after `keepaliveEvery` of quiet. It stops after `keepaliveLongest` (5 minutes) with nothing from the vendor. A translated stream (`streamTranslated`) whose upstream has begun its reply and then says nothing at all, not even a comment (a relay that sends the rest of a long reply in one burst at its end), is kept alive in the client's own protocol by a ticker beside its reader (`keepThroughSilence`): every `keepaliveEvery` it sends the client's keepalive (`enc.keepalive`: a ping for Anthropic, `response.in_progress` for Responses, a comment for Chat, nothing for Gemini), which also counts as the try heard from, so the held stream's comments go on too; it stops once the upstream has been silent for `translatedQuietLongest` (30 minutes), past the 10 minutes `upstreamHeaderWait` gives a vendor to begin and long enough for a long reasoning turn sent whole at its end, after which the client's own timeout ends the wait (youyo.p on Discord: before, Claude Code heard only comments, which its event watchdog doesn't count, and gave up 300s on; `TestTranslatedSilenceKeptAliveForABound`). An upstream that has answered its headers but sent no byte yet is still waited for under the held stream's comments alone (`notAnAPIReply` reads its first bytes before the reply is translated). An Anthropic client is sent a ping at most once each `pingEvery` (1 minute), comments between, and none before `message_start`: Claude Code's watchdog counts only 30 pings in a row as life, and a ping on every upstream keepalive spent them in the first minute; the SDKs' `MessageStream` fails on any event before `message_start` (`anthropicEncoder.keepalive`, `TestTranslatedStreamKeepsClientAlive`). A try waiting for its key's or account's slot or for room in its minute (`lanes.take`, `rpms.wait`) is kept alive the same way (`keepQueued`), for as long as it waits; once the 200 is out, the try is held, and its failure or its queue turning it away reaches the agent as the stream's error (`failTo`), not a status (coeo91 on Discord). A wait shorter than `keepHeldAfter` sends nothing, so its 429 keeps its `Retry-After`. A proxy in front such as Cloudflare (125s) otherwise ends the request (#947). Gemini streams get no comments (#934). A vendor that then ignores `stream:true` and answers 200 with its reply whole has it sent as the agent's stream (`wholeAsStream`: Chat, Responses, Anthropic), not as the stream's error, while a reply under a type that says neither JSON nor a stream is held as a stream when its first bytes are one (`sseStart`), read event by event like one that says so, so an error among its frames still goes to the next candidate (#1418: it went through at once); a 200 that is no reply of the protocol's fails as a translated reply that didn't stream does (502, "… did not stream"), for the next candidate to answer, and the last one's reaches the agent as the stream's error. A held stream that has sent nothing but the frames before a reply (`response.created`, `response.in_progress`, `codex.rate_limits`, pings, an empty part begun, and any event `streamEvent` doesn't know to carry text, reasoning, a call or a reply's end (`replyEvent`), or an item not known that isn't a call or a tool's (`streamPart.said`): `eventLead`; Aliceonly on #1418: an unknown event was taken for content and let a frames-only 200 through, and its `server_is_overloaded` reached Codex with five candidates left) stays held as long as its agent is kept alive, up to `holdLead` (4 minutes, under the 300s Codex waits for an event; `keptQuiet`), so a `server_is_overloaded` that comes after it is a failure the next candidate answers (#1418: let through at 15s, the ChatGPT backend's failure 36s in ended Codex's turn with sub2api, next in the group, never asked). A stream whose agent can't be kept alive (Gemini's) is let through after `holdLongest` (15s), as before. Codex counts only events toward that 300s, not comments, so the bound is the request's, not each try's: once Codex on Responses has heard no event for `agentQuietMost` (4 minutes) since the request began or its last event (`keptAlive.tooQuiet`), the try in hand is let through (`scan`, `keepQuiet`), and no group replan or same-one retry is taken whose pause would end past it (fadenoob on Discord: held tries and replan pauses added up past 300s and Codex reconnected on "idle timeout waiting for SSE"). A translated reply whose upstream stream stops before it said its reply had — an Anthropic one with no stop_reason and no message_stop, a Responses one with no `response.completed`, the same two a relayed one is told by (`sniff.whole`; a Chat stream may end without `[DONE]` and be whole), a Gemini or Code Assist one whose last chunk has no candidate's `finishReason` (`geminiEnds`, #1643: `usageMetadata` is no end, Vertex AI sends it on every chunk; a chunk with no candidate at all is the empty reply's to answer, #667), and a relayed Gemini one told the same way (#1644; it ends in Gemini's own `{"error":{"code":502,…,"status":"UNAVAILABLE"}}` data line from `streamFailure`, `TestGeminiRelayEndedShort`; whatever a relayed stream gets appended, that error or a filled-in `finish_reason`, begins after a blank line, two newlines when the upstream's last write stopped inside a line, so it is an event of its own rather than more of the half-written one, #2446, `TestPassthroughStreamCutMidLine`, `TestGeminiCompatFinishAfterPartialLine`), and a Chat one from Gemini's OpenAI-compatible API (`geminiCompat`) with no `finish_reason`, relayed or translated (`saysItsEnd`; Linka on Discord: the upstream hung up between a whole function call and its `finish_reason`, and Pi threw the call away as "Stream ended without finish_reason"; that API counts the usage only once the generation is done, so when the last chunk carried it only the `finish_reason` was lost and it is filled in, `tool_calls` after a call and `stop` otherwise, as a relayed chunk before `[DONE]` or a translated reply's end, and the reply counts as whole (`chatTail`); only one with no usage at its end is a cut, `TestGeminiCompatRelayEndedShort`, `TestGeminiCompatTranslatedEndedShort`) — ends the turn with "the reply ended before it was complete" (`errEndedShort`, `TestResponsesUpstreamEndedShort`, `TestGeminiUpstreamEndedShort`), asked again in place where the upstream goes on from a prefill (`continuation.go`). A client that asked for no stream is told the same 502 of such a cut (`translate`, `TestNonStreamEndedShort`, `TestGeminiNonStreamEndedShort`), rather than handed what came as its whole reply. A translated reply from an Anthropic or Responses upstream that ended but said nothing (no text, call, search or image) and counted no token at all is no reply either, whatever its end says (`repliesCounted`, #2086: an Anthropic-compatible proxy's 200 ended a Codex turn as `response.completed` with no output, no usage and no served model, and Codex didn't ask again). It is the empty reply's to answer (#667): asked again in place, then failed with "an empty reply", streamed or not, for every client (`TestUncountedEmptyReplyFails`, `TestUncountedEmptyReplyFailsEveryClient`). Claude's own empty `end_turn` counts its prompt and stays a whole reply (`TestWellFormedStreamsStillComplete`). A reply passed through in its own protocol isn't read this way.- A vendor's error keeps its `code` and `param` on its way to an OpenAI-shaped client (`writeVendorError`), where `writeError` used to set both to null: a status error relayed or translated, and a request refused inside a Responses stream. A Responses stream's `error` event or failed response whose error is of type `invalid_request_error` is the request at fault (`requestRefused`, read by its type, not its words): its `Event` carries status 400 and the vendor's error (`Event.Said`), so a client answered whole gets the vendor's 400 with its code and param (the whole-translated path wrote 502 `api_error`), a reply cut by it isn't resumed (`continuation.again`), and a translated stream's error says the vendor's type, code and param (`saidError`), so `holdWriter` reads it as the 400 it is. Before, the hold read magpie's `api_error` as a 502: the same request was sent three times, then to the other accounts, and every one rested (magpie-releases#18: the ChatGPT backend answers `access_programs` naming a program the workspace hasn't got with a 200 stream whose error is `access_program_not_enabled`). Asked once, nobody rests ([`codex_access_programs_test.go`](../../internal/gateway/codex_access_programs_test.go)).- An error that reaches the agent in a stream already begun can't carry its status, so `streamError` says it in the words (`saysPassing`, #2372, zminds): for a status that may pass (408, 429, 5xx) whose text names none of the words a text-judging agent retries on (Pi decides by its `errorMessage` alone), it appends `(<status>: you can retry your request)`; a request's own fault (400, 401, 404 …) and a text that already says so are left as they were. `TestStreamErrorSaysItPasses` holds Pi's own lists; `TestGroupLastFailureInStreamSaysItPasses` runs a group whose last member fails after the keepalive. A 413 byte cap in that stream says `request_too_large` in the error text, and Chat and Anthropic name that type, so Pi compacts; Gemini keeps its own status shape (`TestStreamedByteCapSaysRequestTooLargeToPi`, `TestStreamErrorByteCapInEveryShape`).- Claude's, GPT's and Gemini's reasoning is held (`refusesAfterThinking`, up to `holdThinking`, 4 minutes) so that a refusal after it goes to the next candidate unseen (#248). It is held only while one could: on the last candidate a refusal isn't asked again, so their reasoning streams as it comes there, and a refusal after it reaches the agent as the vendor's own. Held there, a lone relay's Claude or GPT showed Claude Code its thinking all at once with the text, and Codex none until then (iTianbao on X). Gemini's streams there too: a reply that only reasoned is asked again in place by `streamTranslated`, the reasoning the agent has kept and the next try's after it (#667), so holding it only kept Codex on a Gemini model with nothing at all, not even `response.created`, for the whole of its thinking (LuckymanJC on Discord; `TestThinkingStreamsWithNoOneElse`'s gemini cases). Any other model's reasoning is never held.@@ -544,6 +544,9 @@ doesn't count, and hung up after ~315s (#2316). An error before any`KAlive` is still a status another account may answer. Covered by`TestSearchQueueKeepsCodexAlive` in[`search_queue_test.go`](../../internal/gateway/search_queue_test.go).+A provider that says nothing at all, not even a keepalive, is handed on a+`KAlive` every `keepaliveEvery` by the same ticker `streamTranslated` uses+(`keepThroughSilence`), for up to `translatedQuietLongest` of its silence.## Which providers search for another modeldiff --git a/internal/gateway/anthropic.go b/internal/gateway/anthropic.goindex 249a417f7..e953e520a 100644--- a/internal/gateway/anthropic.go+++ b/internal/gateway/anthropic.go@@ -1418,6 +1418,7 @@ type anthropicEncoder struct {// apart from the one open with what it says firstfresh boolstarted bool+ pinged time.Time // the last ping keepalive sentcol collector// cite cuts the text said after a search into lines, each naming a// page found its own block with citations (search_cite.go); cited:@@ -1659,8 +1660,17 @@ func (e *anthropicEncoder) event(ev Event) {}// keepalive is Anthropic's own: a ping event, which its SDKs skip and-// Claude Code's stream watchdog counts.+// Claude Code's stream watchdog counts, at most one each pingEvery+// (Claude Code counts 30 in a row); a comment in between. Before+// message_start it is a comment too: the SDKs' MessageStream fails on+// any event before it ("Unexpected event order, got ping before+// message_start"), and Anthropic sends none.func (e *anthropicEncoder) keepalive() {+ if !e.started || !e.pinged.IsZero() && time.Since(e.pinged) < pingEvery {+ e.w.comment("keepalive")+ return+ }+ e.pinged = time.Now()e.w.event("ping", map[string]any{"type": "ping"})}diff --git a/internal/gateway/continuation.go b/internal/gateway/continuation.goindex 19dcf8075..8c6c6eb6c 100644--- a/internal/gateway/continuation.go+++ b/internal/gateway/continuation.go@@ -26,6 +26,7 @@ import ("net/http""slices""strings"+ "sync""time""unicode"@@ -538,7 +539,20 @@ func (s *Server) streamTranslated(w http.ResponseWriter, r *http.Request, p prov}ended := false // the upstream said its reply endedvar tail chatTail+ // the client is kept alive while the upstream says+ // nothing at all, its first event yet to come too: the+ // upstream's 200 is in, and a holdWriter holds what is+ // written for another candidate to answer still+ var wmu sync.Mutex+ heard, stopKeep := keepThroughSilence(&wmu, func() {+ if failed == "" && !cut && !relooped && sw.quiet() >= keepaliveEvery/2 {+ enc.keepalive()+ }+ })serr = readSSEAlive(rd, func(_, data string) error {+ wmu.Lock()+ defer wmu.Unlock()+ heard()tail.see([]byte(data))st := upstreamStop([]byte(data))u.add(Usage{Upstream: upstreamOf([]byte(data)), Stop: st, Tier: upstreamTier([]byte(data))})@@ -550,7 +564,13 @@ func (s *Server) streamTranslated(w http.ResponseWriter, r *http.Request, p provreturn errStreamCut}return nil- }, alive)+ }, func() {+ wmu.Lock()+ defer wmu.Unlock()+ heard()+ alive()+ })+ stopKeep()res.Body.Close()if errors.Is(serr, errStreamCut) {serr = nildiff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.goindex d17c50d1e..a339ee5cc 100644--- a/internal/gateway/gateway.go+++ b/internal/gateway/gateway.go@@ -6562,6 +6562,73 @@ var keepaliveGap = time.Second// reply stuck longer is left for the client's own idle timeout to end.var keepaliveEvery, keepaliveLongest = 15 * time.Second, 5 * time.Minute+// translatedQuietLongest is how long a translated reply's upstream may+// say nothing at all, its stream open, while magpie keeps the client+// alive every keepaliveEvery (keepThroughSilence). A reply that comes in+// one burst at its end (claude-opus-5-5 at effort max, a relay that+// buffers) is silent for its whole generation, which runs past 10+// minutes, longer than the 10 minutes magpie gives a vendor for its+// headers (upstreamHeaderWait); a dead connection ends sooner by itself+// (HTTP2Health, TCP keepalive). Past it the client's own idle timeout+// ends the wait: Claude Code's, 300s after its last ping counted.+var translatedQuietLongest = 30 * time.Minute++// pingEvery is the least time between two of Anthropic's ping events+// magpie sends one client; a keepalive due sooner is a comment. Claude+// Code (2.1.311) starts its 300s stream watchdog again on a ping only+// for 30 in a row (Gcr): pings a second apart, as magpie sent them+// while an upstream sent its own keepalives, spend those in 30s, and the+// watchdog ends the reply 300s later. One a minute lasts+// translatedQuietLongest,+// and the comments between keep the bytes going for a proxy and for+// Claude Code's byte watchdog.+var pingEvery = time.Minute++// keepThroughSilence keeps a translated reply's client alive while its+// upstream, its 200 and stream begun, says nothing at all: keep is+// called every keepaliveEvery, until stop, while the upstream was last+// heard (heard, at each event or comment it sends) less than+// translatedQuietLongest ago. keep is called holding lock, which the+// caller holds around everything else it writes to the client. Before,+// such a reply sent its client nothing until the upstream spoke, and+// Claude Code's watchdog ended it (youyo.p on Discord).+func keepThroughSilence(lock sync.Locker, keep func()) (heard, stop func()) {+ var mu sync.Mutex+ last := time.Now()+ done, over := make(chan struct{}), make(chan struct{})+ go func() {+ defer close(over)+ tick := time.NewTicker(keepaliveEvery)+ defer tick.Stop()+ for {+ select {+ case <-done:+ return+ case <-tick.C:+ mu.Lock()+ quiet := time.Since(last)+ mu.Unlock()+ if quiet >= translatedQuietLongest {+ continue+ }+ lock.Lock()+ keep()+ lock.Unlock()+ }+ }+ }()+ heard = func() {+ mu.Lock()+ last = time.Now()+ mu.Unlock()+ }+ stop = func() {+ close(done)+ <-over+ }+ return heard, stop+}+// relayEvents hands see each of events until they end or see says stop,// keeping the client of sw alive while none comes (keepaliveEvery, for up// to keepaliveLongest since the last).diff --git a/internal/gateway/keepalive_test.go b/internal/gateway/keepalive_test.goindex e4bcdfd36..9b3083d4e 100644--- a/internal/gateway/keepalive_test.go+++ b/internal/gateway/keepalive_test.go@@ -30,9 +30,17 @@ func keepFast(t *testing.T, gap, every, longest time.Duration) {// (#2169, #2711, #2737). The client must hear something it counts all along: Codex's// timeout counts events, never a comment, so a Responses or Anthropic// client is kept alive with events, a Chat one with comments.+//+// An Anthropic client's pings are spaced pingEvery apart (here 100ms),+// with comments between: Claude Code counts only 30 pings in a row as+// life, so a ping on every provider keepalive (here each 20ms, ~75 in+// all) spent them early and let its watchdog fire later (youyo.p).func TestTranslatedStreamKeepsClientAlive(t *testing.T) {keepFast(t, 10*time.Millisecond, keepaliveEvery, keepaliveLongest)const quietFor, idleFor = 1500 * time.Millisecond, 500 * time.Millisecond+ pe := pingEvery+ pingEvery = 100 * time.Millisecond+ t.Cleanup(func() { pingEvery = pe })for _, c := range []struct {name, path, body, done, ping string // done: the reply's last eventupChat bool@@ -169,49 +177,123 @@ func TestTranslatedStreamKeepsClientAlive(t *testing.T) {if empty > 0 {t.Fatalf("%d events with no data", empty)}+ // one each pingEvery over quietFor, and room for the first+ // and a slow scheduler's catch-up+ if most := int(quietFor/pingEvery) + 5; c.ping == `"type":"ping"` && pings > most {+ t.Fatalf("%d pings in %v, want at most %d (one each %v)", pings, quietFor, most, pingEvery)+ }})}}-// TestTranslatedSilenceStillTimesOut: a provider that says nothing at all-// isn't covered for; the client's idle timeout still ends the wait.-func TestTranslatedSilenceStillTimesOut(t *testing.T) {- keepFast(t, 10*time.Millisecond, keepaliveEvery, keepaliveLongest)- fresh(t)- release := make(chan struct{})- up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- io.Copy(io.Discard, r.Body)- w.Header().Set("Content-Type", "text/event-stream")- io.WriteString(w, "data: "+`{"id":"c1","model":"m1","choices":[{"index":0,"delta":{"content":"progress"},"finish_reason":null}]}`+"\n\n")- w.(http.Flusher).Flush()- select {- case <-r.Context().Done():- case <-release:- }- }))- defer up.Close()- defer close(release)- if err := provider.Save(provider.Provider{ID: "fake", Name: "Fake", Key: "fixture", Models: []string{"m1"}, Chat: up.URL + "/v1"}); err != nil {- t.Fatal(err)- }- server := httptest.NewServer(New().Handler())- defer server.Close()- // The client's idle timeout is over the body: it starts once the- // headers are in, so that a loaded machine's slow first byte isn't- // taken for it (#2248). 400ms is 40 keepalive gaps.- ctx, cancel := context.WithCancel(context.Background())- defer cancel()- req, _ := http.NewRequestWithContext(ctx, "POST", server.URL+"/v1/responses", strings.NewReader(`{"model":"fake/m1","stream":true,"input":"hi"}`))- res, err := http.DefaultClient.Do(req)- if err != nil {- t.Fatal(err)- }- defer res.Body.Close()- idle := time.AfterFunc(400*time.Millisecond, cancel)- defer idle.Stop()- b, _ := io.ReadAll(res.Body)- if n := strings.Count(string(b), `"type":"response.in_progress"`); n > 1 {- t.Fatalf("a silent provider was covered for: %d in_progress events\n%s", n, b)+// TestTranslatedSilenceKeptAliveForABound: a provider that begins its+// reply and then says nothing at all, not even a comment, as a relay does+// that sends the rest of a long reply in one burst at its end (youyo.p on+// Discord: Claude Code on claude-opus-5-5 at effort max, a custom+// provider's /v1/responses, the request canceled some 357s in). The+// client is kept alive in its own protocol, with what its idle timeout+// counts: a ping for an Anthropic client, a response.in_progress for a+// Responses one, a comment for a Chat one. It is kept so only for+// translatedQuietLongest of silence (here 1s), after which the client's+// own idle timeout ends the wait. Before, the client was sent only+// comments, which neither Claude Code's event watchdog nor Codex counts,+// for keepaliveLongest.+func TestTranslatedSilenceKeptAliveForABound(t *testing.T) {+ const every, bound = 20 * time.Millisecond, time.Second+ keepFast(t, 10*time.Millisecond, every, 100*time.Millisecond)+ k, w, q, p, l := keepHeldAfter, watchEvery, agentQuietMost, pingEvery, translatedQuietLongest+ keepHeldAfter, watchEvery, agentQuietMost, pingEvery, translatedQuietLongest = 30*time.Millisecond, 10*time.Millisecond, 200*time.Millisecond, every, bound+ t.Cleanup(func() { keepHeldAfter, watchEvery, agentQuietMost, pingEvery, translatedQuietLongest = k, w, q, p, l })+ for _, c := range []struct {+ name, path, body, alive string+ upChat bool // the provider speaks Chat, else Responses+ }{+ {"anthropic_from_responses", "/v1/messages", `{"model":"fake/m1","stream":true,"max_tokens":64,"messages":[{"role":"user","content":"hi"}]}`, `data: {"type":"ping"}`, false},+ {"responses_from_chat", "/v1/responses", `{"model":"fake/m1","stream":true,"input":"hi"}`, `"type":"response.in_progress"`, true},+ {"chat_from_responses", "/v1/chat/completions", `{"model":"fake/m1","stream":true,"messages":[{"role":"user","content":"hi"}]}`, ": keepalive", false},+ } {+ t.Run(c.name, func(t *testing.T) {+ fresh(t)+ release := make(chan struct{})+ up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ io.Copy(io.Discard, r.Body)+ w.Header().Set("Content-Type", "text/event-stream")+ // the reply begins, then nothing comes on the open stream+ if c.upChat {+ io.WriteString(w, "data: "+`{"id":"c1","model":"m1","choices":[{"index":0,"delta":{"content":"progress"},"finish_reason":null}]}`+"\n\n")+ } else {+ io.WriteString(w, "event: response.created\ndata: "+`{"type":"response.created","response":{"id":"r1","model":"m1"}}`+"\n\n")+ io.WriteString(w, "event: response.output_text.delta\ndata: "+`{"type":"response.output_text.delta","delta":"progress"}`+"\n\n")+ }+ w.(http.Flusher).Flush()+ select {+ case <-r.Context().Done():+ case <-release:+ }+ }))+ defer up.Close()+ defer close(release)+ p := provider.Provider{ID: "fake", Name: "Fake", Key: "fixture", Models: []string{"m1"}}+ if c.upChat {+ p.Chat = up.URL + "/v1"+ } else {+ p.Responses = up.URL + "/v1"+ }+ if err := provider.Save(p); err != nil {+ t.Fatal(err)+ }+ server := httptest.NewServer(New().Handler())+ defer server.Close()+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ req, _ := http.NewRequestWithContext(ctx, "POST", server.URL+c.path, strings.NewReader(c.body))+ req.Header.Set("Content-Type", "application/json")+ res, err := http.DefaultClient.Do(req)+ if err != nil {+ t.Fatal(err)+ }+ defer res.Body.Close()+ lines := make(chan string, 1024)+ go func() {+ defer close(lines)+ sc := bufio.NewScanner(res.Body)+ for sc.Scan() {+ lines <- sc.Text()+ }+ }()+ // Kept alive: the client hears 10 of what it counts, which at+ // one each 20ms takes 200ms of the 1s bound. A loaded machine+ // may take longer, so it is waited for, up to 20s.+ count := func(until <-chan time.Time, enough int) int {+ n := 0+ for {+ select {+ case line, ok := <-lines:+ if !ok {+ return n+ }+ if strings.Contains(line, c.alive) {+ if n++; n == enough {+ return n+ }+ }+ case <-until:+ return n+ }+ }+ }+ if n := count(time.After(20*time.Second), 10); n < 10 {+ t.Fatalf("a silent provider's client heard %d of %q, want it kept alive", n, c.alive)+ }+ // Let go: one each 20ms for the bound is at most ~55 more+ // (the bound, and the held stream's 100ms of comments after+ // it). Unbounded, the 4s after would hold ~200.+ n := count(time.After(4*time.Second), -1)+ t.Logf("%d more of %q after the first 10", n, c.alive)+ if n > 100 {+ t.Fatalf("the client was kept alive %d more times, long past the provider's %v of silence", n, bound)+ }+ })}}@@ -242,7 +324,9 @@ func TestRelayKeepsClientAlive(t *testing.T) {if len(parts) != 2 || !strings.Contains(body, "message_stop") {t.Fatalf("reply incomplete:\n%s", body)}- before, after := strings.Count(parts[0], `"type":"ping"`), strings.Count(parts[1], `"type":"ping"`)+ // a ping or, within pingEvery of the last, a comment+ alive := func(s string) int { return strings.Count(s, `"type":"ping"`) + strings.Count(s, ": keepalive") }+ before, after := alive(parts[0]), alive(parts[1])t.Logf("pings in a 150ms gap: %d, in a 600ms gap (200ms kept): %d", before, after)if before < 3 {t.Fatalf("a quiet reply wasn't kept alive: %d pings in 150ms", before)@@ -277,3 +361,30 @@ func TestTraceKeepsInflight(t *testing.T) {t.Fatalf("%d routes kept, want %d", n, 2*traceKeep)}}++// Anthropic's SDKs fail a stream with any event before message_start+// ("Unexpected event order, got ping before message_start"), and Anthropic+// sends none, so a keepalive before it is a comment; after it a ping, at+// most one each pingEvery, comments in between.+func TestAnthropicKeepaliveBeforeStartIsAComment(t *testing.T) {+ was := pingEvery+ pingEvery = time.Hour+ t.Cleanup(func() { pingEvery = was })+ rec := httptest.NewRecorder()+ e := &anthropicEncoder{w: newSSEWriter(rec), model: "m"}+ e.keepalive()+ e.keepalive()+ if body := rec.Body.String(); strings.Contains(body, "event:") || !strings.Contains(body, ": keepalive") {+ t.Fatalf("before message_start: %q", body)+ }+ e.event(Event{Kind: KStart, MsgID: "msg_1", Model: "m"})+ e.keepalive()+ e.keepalive()+ body := rec.Body.String()+ if !strings.HasPrefix(strings.TrimPrefix(body[strings.Index(body, "event:"):], "event: "), "message_start") {+ t.Fatalf("an event before message_start: %q", body)+ }+ if n := strings.Count(body, `"type":"ping"`); n != 1 {+ t.Fatalf("%d pings within one pingEvery after message_start: %q", n, body)+ }+}diff --git a/internal/gateway/quiet_keepalive_test.go b/internal/gateway/quiet_keepalive_test.goindex 5ccd6b2e9..12c929003 100644--- a/internal/gateway/quiet_keepalive_test.go+++ b/internal/gateway/quiet_keepalive_test.go@@ -184,12 +184,17 @@ func TestSilentHeldStreamStillFailsOver(t *testing.T) {}}-// A vendor quiet for longer than keepaliveLongest is no longer covered-// for: the comments stop, for a stuck stream to end at a timeout.+// A vendor quiet for longer than keepaliveLongest (the held stream's+// comments) and translatedQuietLongest (a translated stream's own,+// keepThroughSilence) is no longer covered for: the comments stop, for a+// stuck stream to end at a timeout.func TestQuietKeptAliveOnlySoLong(t *testing.T) {fresh(t)quietFast(t)keepaliveLongest = 200 * time.Millisecond+ was := translatedQuietLongest+ translatedQuietLongest = 200 * time.Millisecond+ t.Cleanup(func() { translatedQuietLongest = was })never := make(chan struct{})quietOn(t, "fixture", quietCreated+quietDelta("hel"), quietDelta("lo")+quietDone, never)gw := httptest.NewServer(New().Handler())@@ -208,7 +213,10 @@ func TestQuietKeptAliveOnlySoLong(t *testing.T) {if i < 0 || j < i {t.Fatalf("%q", body)}- if n := strings.Count(body[i:j], ": keepalive"); n == 0 || n > 8 {+ // 200ms at one each 40ms is 5, from each of the two keepers, the+ // held stream's and the translated stream's, with room; one that+ // never stopped sends ~75 in the 3s+ if n := strings.Count(body[i:j], ": keepalive"); n == 0 || n > 16 {t.Fatalf("%d keepalives in 3s, kept for 200ms: %q", n, body)}}diff --git a/internal/gateway/search.go b/internal/gateway/search.goindex 8eeae5458..2203e6771 100644--- a/internal/gateway/search.go+++ b/internal/gateway/search.go@@ -1242,7 +1242,20 @@ func (s *Server) askTranslated(p provider.Provider, to provider.Protocol, model}var failed errortold := time.Now() // when the reader was last sent anything+ // the reader hears that the reply goes on while the+ // provider says nothing at all, as streamTranslated's client+ // does (keepThroughSilence)+ var mu sync.Mutex+ heard, stopKeep := keepThroughSilence(&mu, func() {+ if failed == nil && time.Since(told) >= keepaliveEvery/2 {+ failed = send(Event{Kind: KAlive})+ told = time.Now()+ }+ })err := readSSEAlive(rd, func(_, data string) error {+ mu.Lock()+ defer mu.Unlock()+ heard()if err := dec(data, func(ev Event) {if failed == nil {failed = send(ev)@@ -1253,6 +1266,9 @@ func (s *Server) askTranslated(p provider.Provider, to provider.Protocol, model}return failed}, func() {+ mu.Lock()+ defer mu.Unlock()+ heard()// a provider still queued or thinking — keepalives,// response.in_progress — is heard from with nothing to// translate: the reader hears that it goes on (#2316)@@ -1261,6 +1277,7 @@ func (s *Server) askTranslated(p provider.Provider, to provider.Protocol, modeltold = time.Now()}})+ stopKeep()if err != nil && failed == nil && ctx.Err() == nil {send(Event{Kind: KError, Text: p.Name + ": " + err.Error()})}diff --git a/internal/gateway/search_queue_test.go b/internal/gateway/search_queue_test.goindex b2195ddbf..ce18572ec 100644--- a/internal/gateway/search_queue_test.go+++ b/internal/gateway/search_queue_test.go@@ -21,9 +21,9 @@ import (// after the client receives progress, so content cannot hide a lost heartbeat.func TestSearchQueueKeepsCodexAlive(t *testing.T) {keepFast(t, 10*time.Millisecond, 20*time.Millisecond, 100*time.Millisecond)- held, quiet, watch := keepHeldAfter, agentQuietMost, watchEvery- keepHeldAfter, agentQuietMost, watchEvery = 10*time.Millisecond, 50*time.Millisecond, 5*time.Millisecond- t.Cleanup(func() { keepHeldAfter, agentQuietMost, watchEvery = held, quiet, watch })+ held, quiet, watch, silent := keepHeldAfter, agentQuietMost, watchEvery, translatedQuietLongest+ keepHeldAfter, agentQuietMost, watchEvery, translatedQuietLongest = 10*time.Millisecond, 50*time.Millisecond, 5*time.Millisecond, 40*time.Millisecond+ t.Cleanup(func() { keepHeldAfter, agentQuietMost, watchEvery, translatedQuietLongest = held, quiet, watch, silent })for _, mode := range []string{"comments", "progress", "cancel", "silent"} {t.Run(mode, func(t *testing.T) {fresh(t)