Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
gateway: a large body whose h2 ping was lost goes again unpinged (#3909)
yetone05ed578bdc2a
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex 567ce15cb..73b39e6ea 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -38,7 +38,7 @@ backfilled; see [Gateway Session Aggregation](gateway-sessions.md).| Sink | With `Sink` on, a key or account that answers 429 while it still has quota goes to the back, behind every one not rate limited since. Held in memory until restart, or until its rest is lifted by hand (Try now, `Unrest` calls `unsink`, `UnrestAll` `unsinkAll` for the key and each of its models), which puts it back in its place (`TestTryNowPutsASunkAccountBack`, `TestUnrestAllUnsinksTheModels`). Another magpie's model rate limited there, whose rest is the model's alone (`remoteModelRest`), sinks alone too (`restID`), and `sunkAt` reads a candidate's model before its key, so the remote's other models keep their place (#2167, ggbdpq; `TestRemoteMagpieRateLimitSinksTheModel`, `TestMagpieAddedByHandRateLimitSinksTheModel`). A group in a group sinks by its own Sink, within its own order (`sinkOwn`, #2255), and the group asked for then sinks the whole by its own; `TestNestedGroupSinksItsOwnOrder`. | [`sink.go`](../../internal/gateway/sink.go) || Image generation | `/v1/images/generations` and `/v1/images/edits` are translated to the selected model's image API. OpenRouter image-only models use its `/images` endpoint and send reference images as `input_references`; OpenRouter models that return images through chat stay on Chat Completions. Volcengine Ark (`isArk`: an `ark.*.volces.com` base, the preset's plans or one added by hand) is asked at `/images/generations` only, as its doc (https://www.volcengine.com/docs/82379/1541523) takes it (`drawArk`): `watermark: false`, since it marks every image otherwise; reference images as JSON `image` (a URL or data URL, an array for several), never a multipart edit; no `n`, so one request per image (`eachDrawing`); no ratio as `size`, so a ratio its doc lists is sent as that ratio's 2K size and any other is a 400 that says to put it in the prompt; no mask, no quality (JL on Discord, `TestArkDrawsWithoutWatermarkOncePerImage`). A routing group named for a drawing or a video (`drawingGroup`: `group/<id>`, or a bare id the group goes by that resolved to one of its members) is tried member by member in the order its routing plans for a chat (`drawingTries` over `planGroup`), each member over its own keys or accounts (`drawOnAccounts`, `startVideo`); the next member is asked while one fails as a chat's would move on (`retryable`: its vendor 5xx, out of quota, the model not served), never for a refusal of the request itself, and the call's `Fallback` names those tried first (#2005, tianxia3111: a group of two relays answered the first one's 502). Pause rules and a gateway key's group hold apply as for a chat. A failed drawing rests no member and no key. An account out of quota for the image model (`failure` reads `failQuota`) rests that model on it alone (`restDrawer`, keyed by `restID`), until the reset its refusal names (`resetsAt`: Antigravity's "Your quota will reset after 3h19m30s.") or `quotaRest`, so the next drawing asks the accounts that still have it first (`restLast`), and the account's text models stay in. Antigravity also tells a model's quota used up as a 200 whose stream is the error; `drawCodeAssist` gives such an error the status its words stand for (`saidStatus`: quota or rate 429, credit 402, else 502), as the images API's 200 with no image does, so a spent account hands the drawing to the next one as its 429 does (01huadalang on Discord: a 502 ended the drawing on the spent account; `TestAntigravityDrawingOutOfQuotaInsideTheReplyGoesOn`, `TestAntigravityDrawingQuotaRestsTheImageModel`). `TestImageGroupMovesOnToTheNextMember`, `TestVideoGroupMovesOnToTheNextMember`. An account's own list of models (#474) never bars a ChatGPT account's GPT Image, a Grok subscription's Grok Imagine or a WorkBuddy plan's image models (`listsCantName`): the list is picked from the editor's chat models, which they are never among, so a list kept for some chat models had every drawing refused as "set not to be served by any account" (heyiwei5 on X). A key's list, picked from models that include its image ones, still bars them. `TestCodexDrawsPastAccountModelLists`. A drawing is given Settings → Images → Image generation timeout (`imageTimeout`, 1 to 60 minutes, 5 when unset; neighbads on X: a slow image endpoint needed more) as its context's deadline (`drawTimeout`, `drawWithin`), and so is downloading an image answered as a link (`imageLongest`, never under `ImageLongest`). A drawing's request goes through a transport with no `ResponseHeaderTimeout` (`drawsApart`), since an images API sends its headers only once it has drawn, and every other request keeps `upstreamHeaderWait` (10 minutes). A video's submit, status and download calls keep 5 minutes each (`videoTimeout`). The `magpie mcp image` process, which may not read these settings (WSL), waits on a call up to `callLongest` (two hours and a minute), so it never cuts one the gateway still waits on. `TestLongImageTimeoutLetsASlowVendorFinish`, `TestImageTimeoutSettingReachesTheVendor`, `TestOnlyDrawingsWaitPastTheHeaderWait`. | [`draw.go`](../../internal/gateway/draw.go), [`draw_volcengine.go`](../../internal/gateway/draw_volcengine.go), [`video.go`](../../internal/gateway/video.go), [`catalog.go`](../../internal/catalog/catalog.go) || Affinity | A conversation stays with the key or account that answered it, so the vendor's prompt cache is read again. Modes: auto (""), `session`, `turn`, `off`, `strict`. Stored in `affinity.json` next to `providers.json`. A conversation is kept by its id (`conversationID`), and a Claude Code subagent's apart from its lead's by `x-claude-code-agent-id` (`stickKey`): Claude Code sends the session's id with its subagents' requests too, and a subagent's answer took the lead's stick over, so the lead's next turn was judged by the subagent's cache and moved to another account, where its whole conversation was written again (slkassinthehood on X, `TestClaudeConversationKeepsItsAccountsCache`). Behind a remote magpie, the key or account is the remote's key and the provider there that serves the model, its id's first part (`remoteUpstream`, `candidate.vendor`, `sameVendor`): the remote is one key here, but each of its providers keeps its own cache and seals, so a conversation is kept, held strictly, told its sealed reasoning was refused and given its native compaction's reader by that pair, as the same group on the remote does (fremontdango on Discord: a group of one model from two of the remote's providers kept a conversation on the second as the same account when the first was gone, and a strict one failed over to it; `TestRemoteMagpieStrictKeepsTheRemotesProvider`, `TestRemoteMagpieOtherProviderIsNotTheSameAccount`). A remote's group is one vendor by its whole id, its own routing deciding inside it; a model named without a provider is the key alone. Lanes and requests a minute still go by the key here, the limits the user set on it. | [`affinity.go`](../../internal/gateway/affinity.go) |-| One try | `attempt` sends one candidate the request. It goes through the proxy of the account or key it uses, else the provider's (`p.Via`, `ProxyChoice`: `AccountProxies` holds an account's by its name, a key's by its `KeyID`). A request magpie makes for one provider while serving another's (a web search, an image description, a classifier call) takes its own provider's proxy or the global one, never the outer one: `netproxy.With` replaces a choice the context already names, "" too, and `webSearch` starts from none, so the search APIs follow the global proxy (#1522). A provider of the user's set not to check its certificate (`InsecureTLS`, #2383) has it skipped for its own base URLs' hosts alone, through `p.Via` ([Providers and accounts](providers-accounts.md)). Every request to one vendor's host shares one HTTP/2 connection (`upstreamTransport`); one with no frame for 20s is pinged, and dropped when no answer comes within 10s (`netproxy.HTTP2Health`, on `http.DefaultTransport` too), so a connection gone silent without closing (a proxy's way out died, a NAT forgot the flow) fails the requests on it, which move on as a 502 does, and the next request dials a new one. A live vendor answers a ping whatever its streams are doing, so a long wait for a first byte is not cut by it (#2843: a group's requests hung with no first byte until a restart; `TestGroupMovesOnFromSilentHTTP2Connection`). A subscription with its own client is served specially, such as Claude through the genuine Claude Code. The request is relayed as is (`passthrough`) when the model is usable on the agent's API, except that a Claude model where the provider has an Anthropic URL is translated to Messages from Chat or Responses too (`OnMessages`: OpenCode speaks Chat for every model, and a relay drops cache_control there), unless the user set the model's API or the vendor's list names its APIs. An account that only streams (`Account.Stream`: a plugin's, and most subscriptions magpie signs in to) has a request that asked for one body translated, which magpie always asks streamed, except a plugin's on Responses (`gathers`): that is relayed all the same, asked with `stream:true`, and the client given the `response` of the stream's `response.completed` (or `response.incomplete`) as one JSON body (`gathered` in [`gathered.go`](../../internal/gateway/gathered.go); its `output` filled from the stream's `response.output_item.done` items, or else its text, when that event's is empty). So the tools the vendor runs itself, and its response's `tools` and usage details, reach the client (plugins#112: a translation dropped Grok's `x_search`, and Grok answered without searching; `TestGrokPluginNonStreamKeepsHostedTools`). A stream that fails or ends before that event is a 502 (`TestGrokPluginNonStreamEndedShort`), never an empty 200. A Messages request is still translated, its stream having no event that holds the whole reply, and so is a subscription magpie signs in to itself (the ChatGPT backend's, Qoder's): its own code fits its requests. A request offering web search to a provider that doesn't search by itself is translated too, so magpie can search for it (`search.go`; a provider that searches for some of its models only, Command Code's Provider API, lists them in `searchModels` by API, and none count under its `x-cmd-zdr` header, #1590), but on the agent's own API where the provider serves it and no `ResponsesFirst`/`MessagesFirst` rule prefers another: Codex offers web_search on every turn, and each went out as Chat (#997). Otherwise it is translated (`translate`) to the first API `usable` gives: Chat, except Responses for OpenAI's models where their maker serves them (`ResponsesFirst`), Responses for every model at ModelScope, whose Chat API reports neither the tokens its cache answered nor reasoning while its Responses API reports both (`ResponsesTellsCache`, #1578; a 404 there sends the model to Chat, and only there), and Anthropic's Messages for a Claude model where the provider has an Anthropic URL (`MessagesFirst`, which keeps its cache breakpoints, #997). An API that answers the model is not served there (`wrongEndpoint`, `catalog.WrongEndpoint`: the English wordings and AgentRouter's Chinese one, #3766, read escaped too; a model's test reads them as well, see [Providers and accounts](providers-accounts.md)) is remembered in `unfit` until magpie restarts. A Chat API that refuses function tools with reasoning, with no other API to take them to, is asked once more with `reasoning_effort` `none` (#176), unless `none` is known refused (`turnedAway`); `none` refused naming the levels, the tools refusal is the answer, not a retry at the lowest level (#2670). When every API has turned the model away, the API the user picked for it (`ModelAPI`) is asked again, and without a pick each API the provider has a URL for (`Speaks`), Chat first. `Provider.Native`, the API agents with a per-model API (Pi, omp, droid) are wired to ask on, follows the same order. A custom provider with a Gemini URL (#1346) has Gemini last among the APIs it speaks: a Gemini client's request is relayed there, and any other is translated to it when it has no other, or when the user set Gemini as the model's API (`SetModelAPI` takes chat, responses, anthropic or gemini, each only where the provider has its URL; a model set to Gemini has no `Native`, so those agents ask it on their own API and it is translated). Upstream it is always asked `models/{model}:streamGenerateContent?alt=sse` under the base (`upstreamPath`), the model only in the path (`forwardOnce` drops the body's), so a Gemini client that asked for one JSON body is translated, as for Factory's generate route. | [`gateway.go`](../../internal/gateway/gateway.go) |+| One try | `attempt` sends one candidate the request. It goes through the proxy of the account or key it uses, else the provider's (`p.Via`, `ProxyChoice`: `AccountProxies` holds an account's by its name, a key's by its `KeyID`). A request magpie makes for one provider while serving another's (a web search, an image description, a classifier call) takes its own provider's proxy or the global one, never the outer one: `netproxy.With` replaces a choice the context already names, "" too, and `webSearch` starts from none, so the search APIs follow the global proxy (#1522). A provider of the user's set not to check its certificate (`InsecureTLS`, #2383) has it skipped for its own base URLs' hosts alone, through `p.Via` ([Providers and accounts](providers-accounts.md)). Every request to one vendor's host shares one HTTP/2 connection (`upstreamTransport`); one with no frame for 20s is pinged, and dropped when no answer comes within 10s (`netproxy.HTTP2Health`, on `http.DefaultTransport` too), so a connection gone silent without closing (a proxy's way out died, a NAT forgot the flow) fails the requests on it, which move on as a 502 does, and the next request dials a new one. A live vendor answers a ping whatever its streams are doing, so a long wait for a first byte is not cut by it (#2843: a group's requests hung with no first byte until a restart; `TestGroupMovesOnFromSilentHTTP2Connection`). A large body (64KB or more, or of no stated length) is pinged too, as a silent connection takes a body that fits its send buffer, but its ping's answer is queued behind the body's own DATA, so on a slow uplink a working connection loses it (#3909: a ~680KB turn on a 2-10KB/s uplink failed every try). Such a body whose connection was dropped for a lost ping is sent once more, through `GetBody`, on a new connection that is not pinged and is given up on when a write moves no bytes for a minute (`uploadTransport`); for 30 minutes after, large bodies go there at once (`drawsApart.slow`; `TestUploadStillUnderwayIsNotCutByHealthPing`, `TestSlowUplinkSkipsTheLostTry`, `TestLargeBodyOnSilentConnectionIsSentAgain`). A subscription with its own client is served specially, such as Claude through the genuine Claude Code. The request is relayed as is (`passthrough`) when the model is usable on the agent's API, except that a Claude model where the provider has an Anthropic URL is translated to Messages from Chat or Responses too (`OnMessages`: OpenCode speaks Chat for every model, and a relay drops cache_control there), unless the user set the model's API or the vendor's list names its APIs. An account that only streams (`Account.Stream`: a plugin's, and most subscriptions magpie signs in to) has a request that asked for one body translated, which magpie always asks streamed, except a plugin's on Responses (`gathers`): that is relayed all the same, asked with `stream:true`, and the client given the `response` of the stream's `response.completed` (or `response.incomplete`) as one JSON body (`gathered` in [`gathered.go`](../../internal/gateway/gathered.go); its `output` filled from the stream's `response.output_item.done` items, or else its text, when that event's is empty). So the tools the vendor runs itself, and its response's `tools` and usage details, reach the client (plugins#112: a translation dropped Grok's `x_search`, and Grok answered without searching; `TestGrokPluginNonStreamKeepsHostedTools`). A stream that fails or ends before that event is a 502 (`TestGrokPluginNonStreamEndedShort`), never an empty 200. A Messages request is still translated, its stream having no event that holds the whole reply, and so is a subscription magpie signs in to itself (the ChatGPT backend's, Qoder's): its own code fits its requests. A request offering web search to a provider that doesn't search by itself is translated too, so magpie can search for it (`search.go`; a provider that searches for some of its models only, Command Code's Provider API, lists them in `searchModels` by API, and none count under its `x-cmd-zdr` header, #1590), but on the agent's own API where the provider serves it and no `ResponsesFirst`/`MessagesFirst` rule prefers another: Codex offers web_search on every turn, and each went out as Chat (#997). Otherwise it is translated (`translate`) to the first API `usable` gives: Chat, except Responses for OpenAI's models where their maker serves them (`ResponsesFirst`), Responses for every model at ModelScope, whose Chat API reports neither the tokens its cache answered nor reasoning while its Responses API reports both (`ResponsesTellsCache`, #1578; a 404 there sends the model to Chat, and only there), and Anthropic's Messages for a Claude model where the provider has an Anthropic URL (`MessagesFirst`, which keeps its cache breakpoints, #997). An API that answers the model is not served there (`wrongEndpoint`, `catalog.WrongEndpoint`: the English wordings and AgentRouter's Chinese one, #3766, read escaped too; a model's test reads them as well, see [Providers and accounts](providers-accounts.md)) is remembered in `unfit` until magpie restarts. A Chat API that refuses function tools with reasoning, with no other API to take them to, is asked once more with `reasoning_effort` `none` (#176), unless `none` is known refused (`turnedAway`); `none` refused naming the levels, the tools refusal is the answer, not a retry at the lowest level (#2670). When every API has turned the model away, the API the user picked for it (`ModelAPI`) is asked again, and without a pick each API the provider has a URL for (`Speaks`), Chat first. `Provider.Native`, the API agents with a per-model API (Pi, omp, droid) are wired to ask on, follows the same order. A custom provider with a Gemini URL (#1346) has Gemini last among the APIs it speaks: a Gemini client's request is relayed there, and any other is translated to it when it has no other, or when the user set Gemini as the model's API (`SetModelAPI` takes chat, responses, anthropic or gemini, each only where the provider has its URL; a model set to Gemini has no `Native`, so those agents ask it on their own API and it is translated). Upstream it is always asked `models/{model}:streamGenerateContent?alt=sse` under the base (`upstreamPath`), the model only in the path (`forwardOnce` drops the body's), so a Gemini client that asked for one JSON body is translated, as for Factory's generate route. | [`gateway.go`](../../internal/gateway/gateway.go) || Trace | The Routing view's record of each decision, written where the decision is made: candidates, rests, which try answered. `Route.Folder` is the folder the agent says it works in (`workDir`), its last two levels only (`shortFolder`, e.g. `workspace/magpie`), never the full path; a request that says none (a title, a topic check) takes its session's as said last (`trace.sessionFolder`), one without a session id has none (#1068). It is kept in the routing history on disk for as long as the route is (`history.go`: 30 days, 200 MB). The GUI shows it in By session's headings, a request row's tooltip, the stage's agent working in more than one folder, and the tray panel's agent tooltip. | [`trace.go`](../../internal/gateway/trace.go) || Prompt | What each request's prompt held, part by part: the agent's system prompt, tool definitions (an MCP server's tools as one item), instruction files and skills, files its tools read, their other output, and the conversation. `inspectPrompt` reads it from the body as the agent sent it, before translation; `calibrated` scales the estimate to the prompt the vendor counted and sets the window of the model that answered. The route keeps names and sizes only, never the text, so the routing history on disk holds none. For the card's text (masaka on Discord), the trace keeps the last `promptBodies` (10) request bodies in memory only (at most 32 MiB each, 64 MiB together), masked as the vendor got them; for a model let have it unmasked, the body is kept as it went and masked when `PromptText` first reads it, so the request itself never waits on masking (#3973). `PromptText` reads a part's items and their text from that body again (`prompt_text.go`), served to the GUI as `/api/context/text`; once a body is dropped, or magpie quits, its text is gone. It goes on the trace as `Route.Prompt`, with `Route.Conv` naming a conversation that has no session id. The GUI's `/api/context` scores each agent from it (`internal/gui/context.go`). A session's line and its compactions are its own turns': a call with a `Kind`, a Claude Code subagent's (`Route.Sub`, from `x-claude-code-agent-id`), and, in a session whose turns carry tool definitions, a request without any (auto mode's classifier, the title, the quota check) are left out, so a classifier's 40K after a 100K turn is no compaction (`toolSessions`, `TestContextSideCallsAreNoCompaction`). | [`prompt.go`](../../internal/gateway/prompt.go) || Vertex AI | A [Google Vertex AI provider](providers-accounts.md#responsibilities-and-sources-of-truth) speaks Gemini alone. A request goes to `<base>/publishers/google/models/<model>:streamGenerateContent?alt=sse` (`provider.VertexPath`, `upstreamPath`), the model named as the user's upstream name for it says (`UpstreamNameIn`, which the usage log's endpoint names too), with no `model` or `stream` field in its body (`passthrough`, `forwardOnce`, `geminiRequest`). A Gemini client's request is relayed as it came, its own `thinkingConfig`, `safetySettings` and the rest kept: streamed there, asked whole to `:generateContent` (`provider.VertexWholePath`), which answers one JSON body (Factory's rule of translating a whole request, as its route only streams, isn't Vertex AI's). Every other API's is translated (`buildFor`, `buildVertex`) and its stream read by `codeAssistDecoder`, which gives thought parts as reasoning, never as the answer's text. Effort: `vertexEffort` fits it to the model's levels, reasoning turned off going as `minimal` fitted the same way, and holds a model whose levels aren't known to `low`…`high`. `vertexThinking` sends Gemini 3 a `thinkingLevel` (`includeThoughts` false with reasoning off) and 2.5 a `thinkingBudget` in the model's range (`vertexBudgets`): turned off, the budget that stops its thinking, or Pro's least, as Pro can't stop. Thought signatures are every Gemini path's (#1445): a call's signature reaches the client in its id (`signedID`) and goes back on the call it came on, while a call with none in a step Vertex AI signed goes without; a step with nothing to give back has every call sent `skip_thought_signature_validator`. A 400 `Invalid thought signature.` is asked again with every call sent so (`withoutSignatures`), for that request only. A 400 refusing `minimal` (`geminiRefusedLevel`) is asked again at the model's lowest other level (`onEffort`), and `markUnfit(offRefused)` keeps that level until magpie restarts. Both retries are the translated path's. Vertex AI's own Google Search grounding isn't used: `searchHosts` names only `generativelanguage.googleapis.com` for Gemini, so magpie searches for it (`search.go`). | [`vertex.go`](../../internal/gateway/vertex.go), [`gateway.go`](../../internal/gateway/gateway.go), [`codeassist.go`](../../internal/gateway/codeassist.go), [`gemini.go`](../../internal/gateway/gemini.go), [`gemini_signature.go`](../../internal/gateway/gemini_signature.go), [`search.go`](../../internal/gateway/search.go) |diff --git a/internal/gateway/upstream_h2_upload_test.go b/internal/gateway/upstream_h2_upload_test.gonew file mode 100644index 000000000..d779cc881--- /dev/null+++ b/internal/gateway/upstream_h2_upload_test.go@@ -0,0 +1,234 @@+package gateway++import (+ "bytes"+ "context"+ "fmt"+ "io"+ "net"+ "net/http"+ "strings"+ "sync"+ "sync/atomic"+ "testing"+ "time"++ "github.com/yetone/magpie/internal/provider"+)++// slowUplink sits between the gateway and a vendor like a saturated+// uplink: the gateway's bytes get to the vendor at rate bytes a second,+// while the vendor's own pass at once. The gateway's writes still land in+// a socket, as they do on a slow link until its send buffer fills, and the+// vendor hears nothing for as long as the body takes.+type slowUplink struct {+ ln net.Listener+ to string+ rate int+ wg sync.WaitGroup+ mu sync.Mutex+ open []net.Conn+}++func newSlowUplink(t *testing.T, to string, rate int) *slowUplink {+ t.Helper()+ ln, err := net.Listen("tcp", "127.0.0.1:0")+ if err != nil {+ t.Fatal(err)+ }+ p := &slowUplink{ln: ln, to: to, rate: rate}+ p.wg.Add(1)+ go func() {+ defer p.wg.Done()+ for {+ a, err := ln.Accept()+ if err != nil {+ return+ }+ b, err := net.Dial("tcp", to)+ if err != nil {+ a.Close()+ continue+ }+ p.mu.Lock()+ p.open = append(p.open, a, b)+ p.mu.Unlock()+ p.wg.Add(2)+ go p.paced(b, a)+ go p.splat(a, b)+ }+ }()+ t.Cleanup(func() {+ p.ln.Close()+ p.mu.Lock()+ for _, c := range p.open {+ c.Close()+ }+ p.mu.Unlock()+ p.wg.Wait()+ })+ return p+}++// paced carries what the gateway wrote to the vendor a twentieth of a+// second of the uplink at a time.+func (p *slowUplink) paced(dst, src net.Conn) {+ defer p.wg.Done()+ chunk := p.rate / 20+ if chunk < 1 {+ chunk = 1+ }+ buf := make([]byte, chunk)+ for {+ n, err := src.Read(buf)+ if n > 0 {+ if _, werr := dst.Write(buf[:n]); werr != nil {+ return+ }+ time.Sleep(50 * time.Millisecond)+ }+ if err != nil {+ return+ }+ }+}++func (p *slowUplink) splat(dst, src net.Conn) {+ defer p.wg.Done()+ io.Copy(dst, src)+}++// #3909 (yxd-ym): a 175k-token turn (~680KB) on a 2-10KB/s uplink failed+// every try with "http2: client connection lost" after 35-45s, while the+// same uplink answered a small turn. The body is still going out, so the+// vendor sends nothing; the read-idle ping is written behind the body's+// own DATA and can only be answered once the last byte arrives, so the+// health check closed a connection that was working. A body that may still+// be under way is not watched by that ping.+func TestUploadStillUnderwayIsNotCutByHealthPing(t *testing.T) {+ fresh(t)+ body := []byte(`{"model":"gemini-4-argon","messages":[{"role":"user","content":"` ++ strings.Repeat("a", 256<<10) + `"}]}`)+ vendor := h2Vendor(t, func(w http.ResponseWriter, r *http.Request) {+ n, err := io.Copy(io.Discard, r.Body)+ if err != nil {+ t.Errorf("the vendor reading the body: %v", err)+ }+ fmt.Fprintf(w, "%s %d", r.Proto, n)+ })+ // 256KB at 64KB/s: the body is still being delivered for about 4s, and+ // the vendor's frames arrive more than the 0.5s a ping is given here+ // apart, as they do at 2KB/s in the report+ path := newSlowUplink(t, vendor.Listener.Addr().String(), 64<<10)+ quickPings(t, vendor)+ s := New()+ t.Cleanup(s.client.CloseIdleConnections)+ ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)+ defer cancel()+ req, err := http.NewRequestWithContext(ctx, http.MethodPost,+ "https://"+path.ln.Addr().String()+"/v1/chat/completions", bytes.NewReader(body))+ if err != nil {+ t.Fatal(err)+ }+ req.Header.Set("Content-Type", "application/json")+ res, err := s.client.Do(req)+ if err != nil {+ t.Fatalf("a %d-byte body an uplink was still delivering failed: %v; want the vendor's reply", len(body), err)+ }+ defer res.Body.Close()+ got, err := io.ReadAll(res.Body)+ if err != nil {+ t.Fatal(err)+ }+ if want := fmt.Sprintf("HTTP/2.0 %d", len(body)); string(got) != want {+ t.Fatalf("the vendor read %q; want %q", got, want)+ }+}++// After the first lost ping under a large body, the large bodies after it+// go out unpinged at once, without a try lost first: one request reaches+// the vendor per turn.+func TestSlowUplinkSkipsTheLostTry(t *testing.T) {+ fresh(t)+ body := []byte(`{"model":"gemini-4-argon","messages":[{"role":"user","content":"` ++ strings.Repeat("a", 256<<10) + `"}]}`)+ var tries atomic.Int32+ vendor := h2Vendor(t, func(w http.ResponseWriter, r *http.Request) {+ tries.Add(1)+ io.Copy(io.Discard, r.Body)+ io.WriteString(w, r.Proto)+ })+ path := newSlowUplink(t, vendor.Listener.Addr().String(), 64<<10)+ quickPings(t, vendor)+ s := New()+ t.Cleanup(s.client.CloseIdleConnections)+ send := func() error {+ ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)+ defer cancel()+ // the body as the gateway gives a vendor's request one+ req, _ := http.NewRequestWithContext(ctx, http.MethodPost,+ "https://"+path.ln.Addr().String()+"/v1/chat/completions", nil)+ provider.SetBody(req, body)+ defer provider.LetGo(req)+ res, err := s.client.Do(req)+ if err != nil {+ return err+ }+ defer res.Body.Close()+ _, err = io.ReadAll(res.Body)+ return err+ }+ if err := send(); err != nil {+ t.Fatalf("first large body: %v", err)+ }+ if n := tries.Load(); n != 2 {+ t.Fatalf("the vendor saw %d tries of the first body, want 2: one cut by the ping, one unpinged", n)+ }+ if err := send(); err != nil {+ t.Fatalf("second large body: %v", err)+ }+ if n := tries.Load(); n != 3 {+ t.Errorf("the vendor saw %d tries in all, want 3: the second body went unpinged at once", n)+ }+}++// A large body keeps #2843's check: on a connection gone silent, whose+// kernel takes the body whole, it is not left waiting the 10-minute header+// wait, but cut by the ping and sent again on a new connection.+func TestLargeBodyOnSilentConnectionIsSentAgain(t *testing.T) {+ fresh(t)+ body := []byte(`{"model":"m","messages":[{"role":"user","content":"` ++ strings.Repeat("a", 128<<10) + `"}]}`)+ vendor := h2Vendor(t, func(w http.ResponseWriter, r *http.Request) {+ n, _ := io.Copy(io.Discard, r.Body)+ fmt.Fprintf(w, "%s %d", r.Proto, n)+ })+ path := newSilentPath(t, vendor.Listener.Addr().String())+ quickPings(t, vendor)+ s := New()+ t.Cleanup(s.client.CloseIdleConnections)+ u := "https://" + path.ln.Addr().String() + "/v1/chat/completions"+ ask := func() (string, error) {+ ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)+ defer cancel()+ req, _ := http.NewRequestWithContext(ctx, http.MethodPost, u, bytes.NewReader(body))+ res, err := s.client.Do(req)+ if err != nil {+ return "", err+ }+ defer res.Body.Close()+ b, err := io.ReadAll(res.Body)+ return string(b), err+ }+ want := fmt.Sprintf("HTTP/2.0 %d", len(body))+ if got, err := ask(); err != nil || got != want {+ t.Fatalf("first request: %q, %v; want %q", got, err, want)+ }+ path.hush()+ if got, err := ask(); err != nil || got != want {+ t.Fatalf("a large body on the silent connection: %q, %v; want %q on a new connection", got, err, want)+ }+ if n := path.conns(); n != 2 {+ t.Errorf("%d connections dialled, want 2: the silent one and a new one", n)+ }+}diff --git a/internal/gateway/upstream_transport.go b/internal/gateway/upstream_transport.goindex 91a3c76b4..72eab8bac 100644--- a/internal/gateway/upstream_transport.go+++ b/internal/gateway/upstream_transport.go@@ -3,6 +3,8 @@ package gatewayimport ("crypto/tls""net/http"+ "strings"+ "sync/atomic""time""github.com/yetone/magpie/internal/netproxy"@@ -16,7 +18,10 @@ var upstreamTLS *tls.Config// with a proxy of its own is sent through a transport kept for that proxy// (#237), and the ChatGPT backend over HTTP/1.1 (netproxy.HTTP1), a// streamed turn to it over its WebSocket (codexSocket). Every-// other vendor is spoken HTTP/2 to wherever it offers it. A request that+// other vendor is spoken HTTP/2 to wherever it offers it. A large body+// whose connection was dropped for a lost ping goes again through a+// transport that does not ping (#3909), and so do the large bodies after+// it for a while (slowFor). A request that// finds its kept-alive connection dropped is sent again on a new one// (replayable). A vendor is given upstreamHeaderWait to begin answering,// but for a drawing, which an images API answers only once it has drawn:@@ -24,8 +29,10 @@ var upstreamTLS *tls.Config// Image generation timeout gives it (drawWithin), up to 60 minutes.func upstreamTransport() http.RoundTripper {return drawsApart{- any: codexSocket{replayable{netproxy.Dispatch(vendorTransport(upstreamHeaderWait))}},- draws: replayable{netproxy.Dispatch(vendorTransport(0))},+ any: codexSocket{replayable{netproxy.Dispatch(vendorTransport(upstreamHeaderWait))}},+ draws: replayable{netproxy.Dispatch(vendorTransport(0))},+ uploads: codexSocket{replayable{netproxy.Dispatch(uploadTransport(upstreamHeaderWait))}},+ slow: new(atomic.Int64),}}@@ -33,6 +40,23 @@ func upstreamTransport() http.RoundTripper {// request that isn't a drawing.var upstreamHeaderWait = 10 * time.Minute+// uploadBytes is the body size from which a request goes to a vendor over+// a connection the read-idle ping does not watch: on an uplink under about+// 3KB/s a body this big is still being delivered when the ping is due+// (netproxy.PingAfter), and the ping's answer can only come back after the+// last byte, behind the body it was queued behind (netproxy.PingWait after+// it). #3909 sent a 175k-token turn, ~680KB, on a 2-10KB/s uplink.+var uploadBytes int64 = 64 << 10++// slowFor is how long, after a large body's connection was dropped for+// a lost ping, the large bodies after it skip the ping and its lost try.+var slowFor = 30 * time.Minute++// uploadStall is how long a request's write may put no bytes on its+// connection before that connection is given up on: what the read-idle+// ping cannot tell apart from a body still being delivered.+var uploadStall = time.Minute+// vendorTransport is a transport to the vendors that gives one wait to// begin answering, none for 0: the request's context then bounds it.func vendorTransport(wait time.Duration) *http.Transport {@@ -48,15 +72,71 @@ func vendorTransport(wait time.Duration) *http.Transport {}}+// uploadTransport is vendorTransport for a request whose body may still be+// under way when the connection has had no frame for a while: an HTTP/2+// ping is written behind that body's own DATA, so its answer comes only+// after the last byte a slow uplink is still delivering, and the health+// check closed a connection that was working (#3909). No ping here; a+// write that moves no bytes ends the connection (WriteByteTimeout), and a+// vendor that never begins answering is ended by the response header wait.+func uploadTransport(wait time.Duration) *http.Transport {+ t := vendorTransport(wait)+ t.HTTP2 = &http.HTTP2Config{WriteByteTimeout: uploadStall}+ return t+}++// uploading says r's body can outlast the vendor's silence: one of a+// length nobody stated (written as it is read), or one over uploadBytes.+func uploading(r *http.Request) bool {+ return r.Body != nil && r.Body != http.NoBody &&+ (r.ContentLength < 0 || r.ContentLength >= uploadBytes)+}+// drawsApart sends a drawing's request (drawWaitOf) through draws, every-// other through any.-type drawsApart struct{ any, draws http.RoundTripper }+// other through any, but for a large body (uploading) on an uplink found+// slow: through uploads.+//+// The ping stays on a large body by default, as a silent connection+// (#2843) takes a body that fits its send buffer and answers nothing for+// the 10-minute header wait. When the ping is lost under a large body, the+// body may still have been on its way (#3909), so it is sent once more+// through uploads, on a new connection, and the large bodies until+// slowFor has passed go there first. A connection that really was dead+// costs nothing more: the new one is dialled either way.+type drawsApart struct {+ any, draws, uploads http.RoundTripper+ // slow is when the uplink was last found slow, in Unix nanoseconds.+ slow *atomic.Int64+}func (t drawsApart) RoundTrip(r *http.Request) (*http.Response, error) {if _, ok := drawWaitOf(r.Context()); ok {return t.draws.RoundTrip(r)}- return t.any.RoundTrip(r)+ if !uploading(r) || t.slow == nil {+ return t.any.RoundTrip(r)+ }+ if time.Since(time.Unix(0, t.slow.Load())) < slowFor {+ return t.uploads.RoundTrip(r)+ }+ res, err := t.any.RoundTrip(r)+ if err == nil || !lostPing(err) || r.GetBody == nil || r.Context().Err() != nil {+ return res, err+ }+ body, gerr := r.GetBody()+ if gerr != nil {+ return nil, err+ }+ t.slow.Store(time.Now().UnixNano())+ again := r.Clone(r.Context())+ again.Body = body+ return t.uploads.RoundTrip(again)+}++// lostPing says err is Go's HTTP/2 transport closing a connection whose+// health ping went unanswered (closeForLostPing).+func lostPing(err error) bool {+ return strings.Contains(err.Error(), "http2: client connection lost")}// replayable lets Go's transport send a request again, at once and on a new