Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

gateway: another magpie's rate-limited model sinks alone, not the whole remote (#2167)

yetone84ef22dd3b0a
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index 1f4fb096f..98848fef4 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -35,7 +35,7 @@ backfilled; see [Gateway Session Aggregation](gateway-sessions.md).
| Candidates | `plan` lists one provider's candidates: one per key or account on, then each `Fallback` model, with resting ones last (`restLast`). `planGroup` and `planLevel` weigh a group's members together. A nested group plans by its own routing (#576). So does a member whose provider routes other than Smart (`ownRouting`): weighed as one by its first not resting (`firstAwake`), tried whole in its own order; `leastHeld` leaves such a group's order alone too. A provider on Smart, or routed as the group is, is pooled with the group's other keys and accounts. `group_own_routing_test.go`. A member's own fallbacks are not the group's. | [`fallback.go`](../../internal/gateway/fallback.go) |
| Order of keys and accounts | `Routing`: smart (""), `order`, `rotate`, `usage` (least used, with an `usageHalfLife` of 1h), `pace`, `weight`, `free` (free first) or `fast` (speed first). `weighRouted` reads each candidate into a `routes.Seat` and asks the strategy serving its routing (`routes.Find`, the router strategy point of [the plugin runtime](plugin-runtime.md#gateway-extension-points-p3-router-strategy)) for the order: Smart is `routes.Smart`, the core's; the others are `routes.Builtin`, registered by the built-in plugin `rotation`. With it off a routing no strategy serves is tried in its listed order, and `routingOf` treats it as `order` wherever the core asks about a routing (a turn's mark, `goesRound`, `sinks`, a rotate group's sessions, a group's heads). Smart, `usage` and `pace` weigh a subscription by its allowance (`allowances`), and a key by its own usage windows when magpie reads them (`keyAllowance`: a sub2api key given a 5-hour, day or 7-day limit, or a key bought as a plan, such as Kimi Code's or a GLM Coding Plan's); any other key has none. `free` weighs by what a model costs there and nothing else: only a strategy with `routes.Strategy.ByFree` set (`routes.FreeFirst` is the one) has `weighRouted` fill `routes.Seat.Free`, from `provider.FreeModel` — the vendor's own `catalog.Model.Free` word, else a list price of $0 on every count — so a pool routed any other way never reads a model list for it, and one routed `free` spends the members whose model costs nothing before those that spend the allowance or credits, each tier in its listed order (nothing free is left behind a paid one, and a pool with no free model behaves as `order`). A group routed `free` weighs its members whole, each over its own accounts or keys as its provider routes them. `fast` weighs by speed lately and nothing else: only a strategy with `routes.Strategy.BySpeed` set (`routes.Fastest`) has `weighRouted` fill `routes.Seat.TTFT` and `TPS`, from `speedOf` ([`speed.go`](../../internal/gateway/speed.go)): per `restID` (account or key, and model), a decayed average (half-life 20m) of a streamed answer's ms to its first content and of its output tokens a second as `usage.DecodeOf` counts them, written once as an answer ends (`noteTrySpeed`, beside `servedCandidate`) and read with one map lookup per candidate, never from the usage history; past an hour since its last answer a seat counts as not measured, the records are capped at 4096, and a restart forgets them. `routes.Fastest` ranks the awake by their first token plus 200 tokens at their pace (a seat with no pace at the measured seats' middle pace), keeps seats within a tenth of their band's fastest in listed order, and of a pool's requests with seats both measured and not, gives every fifth to one not measured (round the unmeasured, `probes`, so one is never starved); resting seats go after the awake, used-up ones (`UsedShare`) after those, and rests and failover are as for any routing. The trace's rows carry `lateTtft`/`lateTps` for the Routing page's story ("answered fastest lately — first token in 0.8s"). A non-streamed reply tells nothing (no first token), so a pool that never streams stays in listed order bar the probes. A group routed `weight` (#1910) takes its members, not their keys, by smooth weighted round robin: `planLevel` makes each model member (its keys ordered by its own provider's routing) and each nested group one unit, and `weightedUnits` puts the round's pick (`routes.Round` over `group/<id>`, keyed by the member's spelling, `Group.Weights`, 1 if unset) first and the rest in member order for failover; a unit whose head rests leaves the round, while one whose accounts are only near their limit (Smart's 90% band, not resting) keeps its share, the band ordering only its own accounts (`TestWeightedGroupGivesALowGroupItsShare`; Creat.wu on Discord read the Routing page's "kept for when the others can't" as the member passed over, so the page now tells a member behind another by a group's weights or order as such, `memberSplit` in the routing plugin's `client.js`), and a kept conversation moves the round's charge to the member that kept it (`wrrSeat`, `routes.MoveRound`). A lone key or account is weighed against nobody, but what it has left is still read into the trace (`weighRouted`), so the Routing page shows it as it does for several (#1016); all but used up, it keeps its conversation rather than being said to move (`affine`). A failure rests a candidate (`failure`, `failureOf`). | [`routing.go`](../../internal/gateway/routing.go), [`weighted.go`](../../internal/gateway/weighted.go), [`internal/routes/`](../../internal/routes), [`internal/provider/routing.go`](../../internal/provider/routing.go) |
| Limits | Each key or account (a candidate's `who()`) has at most `MaxConcurrency` requests out at once (its own `AccountConcurrency` over it); one more waits in its lane's queue, bounded by `QueueLimit` and `QueueWait` and turned away with a 429 past either (`lanes.take`). `MaxRPM` (coeo91 on Discord: OpenRouter's free models take 20 a minute) bounds how many requests it sends the vendor in any rolling 60 seconds (`rpms`), counting each for a second more (`rpmSlack`), since the vendor counts a request when it arrives and the first on a new connection arrives later by its dial: one more waits for room in the minute, in the order they came, after it has its slot and right before it is sent. A wait that would be longer than 2 minutes (or the provider's `QueueWait`, when shorter: `rpmWait`) is turned away at once with a 429 whose `Retry-After` is when the minute has room; nothing rests for it. An agent gone while it waits gives its place back. Every request that reaches the vendor counts: each try of the candidate loop (a retry, the next key, a fallback) waits in `serve`, and the requests a try makes on the way and the gateway's own asks (count_tokens, System One's decisions, embeddings, drawing, Kimi's search) are counted by `rpmTransport` on `s.client` (`metered`, `paidFor`). count_tokens never waits: a candidate with no room is skipped, and the local estimate answers. A full lane or a minute with no room gives way to a ready key or account of the provider or the group (`laneMate`, `ready`); a fallback is never asked for want of either. A plugin's account sends through its own transport, so only its tries are counted, not a plugin's own extra requests. Set in the provider editor (Concurrency, Requests per minute, Queue size, Queue wait) or with `magpie provider account-concurrency`, `queue` and `rpm`; a subscription's are kept with its picks, and every account of it keeps them, those on beside the first as well (`withPicks`, called by `All` and `AlsoOn`; #1899: a second WorkBuddy AI account had none, and queued 62s with the wait set to 1s). | [`concurrency.go`](../../internal/gateway/concurrency.go), [`rpm.go`](../../internal/gateway/rpm.go), [`internal/provider/concurrency.go`](../../internal/provider/concurrency.go) |
-| Sink | With `Sink` on, a key or account that answers 429 while it still has quota goes to the back, behind every one not rate limited since. Held in memory until restart, or until its rest is lifted by hand (Try now, `Unrest`/`UnrestAll` call `unsink`), which puts it back in its place (`TestTryNowPutsASunkAccountBack`). A group in a group sinks by its own Sink, within its own order (`sinkOwn`, #2255), and the group asked for then sinks the whole by its own; `TestNestedGroupSinksItsOwnOrder`. | [`sink.go`](../../internal/gateway/sink.go) |
+| Sink | With `Sink` on, a key or account that answers 429 while it still has quota goes to the back, behind every one not rate limited since. Held in memory until restart, or until its rest is lifted by hand (Try now, `Unrest` calls `unsink`, `UnrestAll` `unsinkAll` for the key and each of its models), which puts it back in its place (`TestTryNowPutsASunkAccountBack`, `TestUnrestAllUnsinksTheModels`). Another magpie's model rate limited there, whose rest is the model's alone (`remoteModelRest`), sinks alone too (`restID`), and `sunkAt` reads a candidate's model before its key, so the remote's other models keep their place (#2167, ggbdpq; `TestRemoteMagpieRateLimitSinksTheModel`, `TestMagpieAddedByHandRateLimitSinksTheModel`). A group in a group sinks by its own Sink, within its own order (`sinkOwn`, #2255), and the group asked for then sinks the whole by its own; `TestNestedGroupSinksItsOwnOrder`. | [`sink.go`](../../internal/gateway/sink.go) |
| Image generation | `/v1/images/generations` and `/v1/images/edits` are translated to the selected model's image API. OpenRouter image-only models use its `/images` endpoint and send reference images as `input_references`; OpenRouter models that return images through chat stay on Chat Completions. Volcengine Ark (`isArk`: an `ark.*.volces.com` base, the preset's plans or one added by hand) is asked at `/images/generations` only, as its doc (https://www.volcengine.com/docs/82379/1541523) takes it (`drawArk`): `watermark: false`, since it marks every image otherwise; reference images as JSON `image` (a URL or data URL, an array for several), never a multipart edit; no `n`, so one request per image (`eachDrawing`); no ratio as `size`, so a ratio its doc lists is sent as that ratio's 2K size and any other is a 400 that says to put it in the prompt; no mask, no quality (JL on Discord, `TestArkDrawsWithoutWatermarkOncePerImage`). A routing group named for a drawing or a video (`drawingGroup`: `group/<id>`, or a bare id the group goes by that resolved to one of its members) is tried member by member in the order its routing plans for a chat (`drawingTries` over `planGroup`), each member over its own keys or accounts (`drawOnAccounts`, `startVideo`); the next member is asked while one fails as a chat's would move on (`retryable`: its vendor 5xx, out of quota, the model not served), never for a refusal of the request itself, and the call's `Fallback` names those tried first (#2005, tianxia3111: a group of two relays answered the first one's 502). Pause rules and a gateway key's group hold apply as for a chat. A failed drawing rests no member and no key. An account out of quota for the image model (`failure` reads `failQuota`) rests that model on it alone (`restDrawer`, keyed by `restID`), until the reset its refusal names (`resetsAt`: Antigravity's "Your quota will reset after 3h19m30s.") or `quotaRest`, so the next drawing asks the accounts that still have it first (`restLast`), and the account's text models stay in. Antigravity also tells a model's quota used up as a 200 whose stream is the error; `drawCodeAssist` gives such an error the status its words stand for (`saidStatus`: quota or rate 429, credit 402, else 502), as the images API's 200 with no image does, so a spent account hands the drawing to the next one as its 429 does (01huadalang on Discord: a 502 ended the drawing on the spent account; `TestAntigravityDrawingOutOfQuotaInsideTheReplyGoesOn`, `TestAntigravityDrawingQuotaRestsTheImageModel`). `TestImageGroupMovesOnToTheNextMember`, `TestVideoGroupMovesOnToTheNextMember`. An account's own list of models (#474) never bars a ChatGPT account's GPT Image, a Grok subscription's Grok Imagine or a WorkBuddy plan's image models (`listsCantName`): the list is picked from the editor's chat models, which they are never among, so a list kept for some chat models had every drawing refused as "set not to be served by any account" (heyiwei5 on X). A key's list, picked from models that include its image ones, still bars them. `TestCodexDrawsPastAccountModelLists`. A drawing is given Settings → Images → Image generation timeout (`imageTimeout`, 1 to 60 minutes, 5 when unset; neighbads on X: a slow image endpoint needed more) as its context's deadline (`drawTimeout`, `drawWithin`), and so is downloading an image answered as a link (`imageLongest`, never under `ImageLongest`). A drawing's request goes through a transport with no `ResponseHeaderTimeout` (`drawsApart`), since an images API sends its headers only once it has drawn, and every other request keeps `upstreamHeaderWait` (10 minutes). A video's submit, status and download calls keep 5 minutes each (`videoTimeout`). The `magpie mcp image` process, which may not read these settings (WSL), waits on a call up to `callLongest` (two hours and a minute), so it never cuts one the gateway still waits on. `TestLongImageTimeoutLetsASlowVendorFinish`, `TestImageTimeoutSettingReachesTheVendor`, `TestOnlyDrawingsWaitPastTheHeaderWait`. | [`draw.go`](../../internal/gateway/draw.go), [`draw_volcengine.go`](../../internal/gateway/draw_volcengine.go), [`video.go`](../../internal/gateway/video.go), [`catalog.go`](../../internal/catalog/catalog.go) |
| Affinity | A conversation stays with the key or account that answered it, so the vendor's prompt cache is read again. Modes: auto (""), `session`, `turn`, `off`, `strict`. Stored in `affinity.json` next to `providers.json`. A conversation is kept by its id (`conversationID`), and a Claude Code subagent's apart from its lead's by `x-claude-code-agent-id` (`stickKey`): Claude Code sends the session's id with its subagents' requests too, and a subagent's answer took the lead's stick over, so the lead's next turn was judged by the subagent's cache and moved to another account, where its whole conversation was written again (slkassinthehood on X, `TestClaudeConversationKeepsItsAccountsCache`). Behind a remote magpie, the key or account is the remote's key and the provider there that serves the model, its id's first part (`remoteUpstream`, `candidate.vendor`, `sameVendor`): the remote is one key here, but each of its providers keeps its own cache and seals, so a conversation is kept, held strictly, told its sealed reasoning was refused and given its native compaction's reader by that pair, as the same group on the remote does (fremontdango on Discord: a group of one model from two of the remote's providers kept a conversation on the second as the same account when the first was gone, and a strict one failed over to it; `TestRemoteMagpieStrictKeepsTheRemotesProvider`, `TestRemoteMagpieOtherProviderIsNotTheSameAccount`). A remote's group is one vendor by its whole id, its own routing deciding inside it; a model named without a provider is the key alone. Lanes and requests a minute still go by the key here, the limits the user set on it. | [`affinity.go`](../../internal/gateway/affinity.go) |
| One try | `attempt` sends one candidate the request. It goes through the proxy of the account or key it uses, else the provider's (`p.Via`, `ProxyChoice`: `AccountProxies` holds an account's by its name, a key's by its `KeyID`). A request magpie makes for one provider while serving another's (a web search, an image description, a classifier call) takes its own provider's proxy or the global one, never the outer one: `netproxy.With` replaces a choice the context already names, "" too, and `webSearch` starts from none, so the search APIs follow the global proxy (#1522). A provider of the user's set not to check its certificate (`InsecureTLS`, #2383) has it skipped for its own base URLs' hosts alone, through `p.Via` ([Providers and accounts](providers-accounts.md)). Every request to one vendor's host shares one HTTP/2 connection (`upstreamTransport`); one with no frame for 20s is pinged, and dropped when no answer comes within 10s (`netproxy.HTTP2Health`, on `http.DefaultTransport` too), so a connection gone silent without closing (a proxy's way out died, a NAT forgot the flow) fails the requests on it, which move on as a 502 does, and the next request dials a new one. A live vendor answers a ping whatever its streams are doing, so a long wait for a first byte is not cut by it (#2843: a group's requests hung with no first byte until a restart; `TestGroupMovesOnFromSilentHTTP2Connection`). A subscription with its own client is served specially, such as Claude through the genuine Claude Code. The request is relayed as is (`passthrough`) when the model is usable on the agent's API, except that a Claude model where the provider has an Anthropic URL is translated to Messages from Chat or Responses too (`OnMessages`: OpenCode speaks Chat for every model, and a relay drops cache_control there), unless the user set the model's API or the vendor's list names its APIs. An account that only streams (`Account.Stream`: a plugin's, and most subscriptions magpie signs in to) has a request that asked for one body translated, which magpie always asks streamed, except a plugin's on Responses (`gathers`): that is relayed all the same, asked with `stream:true`, and the client given the `response` of the stream's `response.completed` (or `response.incomplete`) as one JSON body (`gathered` in [`gathered.go`](../../internal/gateway/gathered.go); its `output` filled from the stream's `response.output_item.done` items, or else its text, when that event's is empty). So the tools the vendor runs itself, and its response's `tools` and usage details, reach the client (plugins#112: a translation dropped Grok's `x_search`, and Grok answered without searching; `TestGrokPluginNonStreamKeepsHostedTools`). A stream that fails or ends before that event is a 502 (`TestGrokPluginNonStreamEndedShort`), never an empty 200. A Messages request is still translated, its stream having no event that holds the whole reply, and so is a subscription magpie signs in to itself (the ChatGPT backend's, Qoder's): its own code fits its requests. A request offering web search to a provider that doesn't search by itself is translated too, so magpie can search for it (`search.go`; a provider that searches for some of its models only, Command Code's Provider API, lists them in `searchModels` by API, and none count under its `x-cmd-zdr` header, #1590), but on the agent's own API where the provider serves it and no `ResponsesFirst`/`MessagesFirst` rule prefers another: Codex offers web_search on every turn, and each went out as Chat (#997). Otherwise it is translated (`translate`) to the first API `usable` gives: Chat, except Responses for OpenAI's models where their maker serves them (`ResponsesFirst`), Responses for every model at ModelScope, whose Chat API reports neither the tokens its cache answered nor reasoning while its Responses API reports both (`ResponsesTellsCache`, #1578; a 404 there sends the model to Chat, and only there), and Anthropic's Messages for a Claude model where the provider has an Anthropic URL (`MessagesFirst`, which keeps its cache breakpoints, #997). An API that answers the model is not served there (`wrongEndpoint`, `catalog.WrongEndpoint`: the English wordings and AgentRouter's Chinese one, #3766, read escaped too; a model's test reads them as well, see [Providers and accounts](providers-accounts.md)) is remembered in `unfit` until magpie restarts. A Chat API that refuses function tools with reasoning, with no other API to take them to, is asked once more with `reasoning_effort` `none` (#176), unless `none` is known refused (`turnedAway`); `none` refused naming the levels, the tools refusal is the answer, not a retry at the lowest level (#2670). When every API has turned the model away, the API the user picked for it (`ModelAPI`) is asked again, and without a pick each API the provider has a URL for (`Speaks`), Chat first. `Provider.Native`, the API agents with a per-model API (Pi, omp, droid) are wired to ask on, follows the same order. A custom provider with a Gemini URL (#1346) has Gemini last among the APIs it speaks: a Gemini client's request is relayed there, and any other is translated to it when it has no other, or when the user set Gemini as the model's API (`SetModelAPI` takes chat, responses, anthropic or gemini, each only where the provider has its URL; a model set to Gemini has no `Native`, so those agents ask it on their own API and it is translated). Upstream it is always asked `models/{model}:streamGenerateContent?alt=sse` under the base (`upstreamPath`), the model only in the path (`forwardOnce` drops the body's), so a Gemini client that asked for one JSON body is translated, as for Factory's generate route. | [`gateway.go`](../../internal/gateway/gateway.go) |
diff --git a/internal/gateway/remote_magpie_rest_test.go b/internal/gateway/remote_magpie_rest_test.go
index 80fc9a5c4..27791db81 100644
--- a/internal/gateway/remote_magpie_rest_test.go
+++ b/internal/gateway/remote_magpie_rest_test.go
@@ -6,7 +6,9 @@ import (
"net/http"
"net/http/httptest"
"strings"
+ "sync"
"testing"
+ "time"
"github.com/yetone/magpie/internal/provider"
)
@@ -79,3 +81,146 @@ func TestRemoteMagpieQuotaRestsTheModel(t *testing.T) {
t.Fatalf("the group: %d %s; the remote was asked %v, other %v", code, body, asked, other.tried)
}
}
+
+// One of a remote magpie's models rate limited with quota left sinks only
+// its own rest (restID, as its rest is keyed): sinking the remote's whole
+// key sends its other models, which were never limited, behind the next
+// provider too, the same mistake resting the whole remote was.
+func TestRemoteMagpieRateLimitSinksTheModel(t *testing.T) {
+ fresh(t)
+ forgetSinks(t)
+ var mu sync.Mutex
+ var asked []string
+ remote := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ var req struct {
+ Model string `json:"model"`
+ }
+ json.NewDecoder(r.Body).Decode(&req)
+ mu.Lock()
+ asked = append(asked, req.Model)
+ mu.Unlock()
+ w.Header().Set("Content-Type", "application/json")
+ if req.Model == "glm/glm-5" {
+ w.WriteHeader(http.StatusTooManyRequests)
+ io.WriteString(w, `{"error":{"message":"rate limit exceeded, slow down"}}`)
+ return
+ }
+ io.WriteString(w, `{"id":"x","choices":[{"index":0,"message":{"role":"assistant","content":"from `+req.Model+`"},"finish_reason":"stop"}]}`)
+ }))
+ t.Cleanup(remote.Close)
+ if err := provider.Save(provider.Provider{ID: "office", Name: "Office", Preset: provider.RemoteMagpiePreset, Key: "k",
+ Chat: remote.URL + "/v1", Models: []string{"glm/glm-5", "kimi/kimi-k3"}}); err != nil {
+ t.Fatal(err)
+ }
+ other := &keyed{}
+ serveOn(t, "other", "ko", []string{"m"}, other)
+ // glm's rate limit, with quota left, sinks it (the group's Sink puts
+ // what sank behind the others)
+ if err := provider.SaveGroup(provider.Group{Name: "R", Members: []string{"office/glm/glm-5", "office/kimi/kimi-k3"}, Routing: provider.Ordered, Sink: true}); err != nil {
+ t.Fatal(err)
+ }
+ s := New()
+ if code, body := postAs(t, s, "", `{"model":"group/r","messages":[{"role":"user","content":"hi"}]}`); code != 200 || !strings.Contains(body, "from kimi/kimi-k3") {
+ t.Fatalf("glm on the remote: %d %s", code, body)
+ }
+ // the watched group: the remote's kimi, never limited, keeps its place
+ // ahead of a provider that never served this conversation
+ if err := provider.SaveGroup(provider.Group{Name: "G", Members: []string{"office/kimi/kimi-k3", "other/m"}, Routing: provider.Ordered, Sink: true}); err != nil {
+ t.Fatal(err)
+ }
+ mu.Lock()
+ asked = nil
+ mu.Unlock()
+ code, body := postAs(t, s, "", `{"model":"group/g","messages":[{"role":"user","content":"hi"}]}`)
+ if code != 200 || !strings.Contains(body, "from kimi/kimi-k3") || len(other.tried) != 0 {
+ t.Fatalf("the group: %d %s; the remote was asked %v, other %v", code, body, asked, other.tried)
+ }
+}
+
+// The same for a magpie added by hand, as a plain OpenAI-compatible
+// endpoint: its reply names the provider there (providerHeader), so its
+// model's 429 rests that model alone (remoteModelRest) and sinks it
+// alone, and it is that model, not the endpoint's others, that the next
+// Sink order sends to the back. Read at the model's own sink key, not only
+// for the remote magpie preset, or it sinks nothing at all.
+func TestMagpieAddedByHandRateLimitSinksTheModel(t *testing.T) {
+ fresh(t)
+ forgetSinks(t)
+ var mu sync.Mutex
+ var asked []string
+ remote := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ var req struct {
+ Model string `json:"model"`
+ }
+ json.NewDecoder(r.Body).Decode(&req)
+ mu.Lock()
+ asked = append(asked, req.Model)
+ mu.Unlock()
+ w.Header().Set("Content-Type", "application/json")
+ if req.Model == "glm/glm-5" {
+ w.Header().Set(providerHeader, "glm")
+ w.WriteHeader(http.StatusTooManyRequests)
+ io.WriteString(w, `{"error":{"message":"rate limit exceeded, slow down"}}`)
+ return
+ }
+ io.WriteString(w, `{"id":"x","choices":[{"index":0,"message":{"role":"assistant","content":"from `+req.Model+`"},"finish_reason":"stop"}]}`)
+ }))
+ t.Cleanup(remote.Close)
+ if err := provider.Save(provider.Provider{ID: "rm", Name: "Remote Magpie", Key: "k",
+ Chat: remote.URL + "/v1", Models: []string{"glm/glm-5", "kimi/kimi-k3"}}); err != nil {
+ t.Fatal(err)
+ }
+ other := &keyed{}
+ serveOn(t, "other", "ko", []string{"m"}, other)
+ s := New()
+ group := func(name string, members ...string) string {
+ t.Helper()
+ if err := provider.SaveGroup(provider.Group{Name: name, Members: members, Routing: provider.Ordered, Sink: true}); err != nil {
+ t.Fatal(err)
+ }
+ mu.Lock()
+ asked = nil
+ mu.Unlock()
+ other.tried = nil
+ code, body := postAs(t, s, "", `{"model":"group/`+strings.ToLower(name)+`","messages":[{"role":"user","content":"hi"}]}`)
+ if code != 200 {
+ t.Fatalf("group %s: %d %s", name, code, body)
+ }
+ return body
+ }
+ if body := group("R", "rm/glm/glm-5", "rm/kimi/kimi-k3"); !strings.Contains(body, "from kimi/kimi-k3") {
+ t.Fatalf("glm there: %s", body)
+ }
+ // kimi there, never limited, keeps its place
+ if body := group("K", "rm/kimi/kimi-k3", "other/m"); !strings.Contains(body, "from kimi/kimi-k3") || len(other.tried) != 0 {
+ t.Errorf("kimi sank with glm: %s; other asked %v", body, other.tried)
+ }
+ // glm, limited there, went to the back: its rest over, it is still
+ // behind other
+ restingUntil.Lock()
+ delete(restingUntil.m, "rm/glm/glm-5")
+ delete(restingUntil.note, "rm/glm/glm-5")
+ restingUntil.Unlock()
+ if body := group("L", "rm/glm/glm-5", "other/m"); !strings.Contains(body, "from ko") || len(asked) != 0 {
+ t.Errorf("glm didn't sink: %s; the remote was asked %v", body, asked)
+ }
+}
+
+// UnrestAll lifts a key's own rest and each of its models', and with them
+// each one's sink: a model that sank alone comes back to the front too.
+func TestUnrestAllUnsinksTheModels(t *testing.T) {
+ forgetSinks(t)
+ now := time.Now()
+ sink("office/glm/glm-5", now)
+ sink("office", now)
+ sink("officer/glm/glm-5", now)
+ New().UnrestAll("office")
+ for _, k := range []string{"office/glm/glm-5", "office"} {
+ if _, ok := sunk.m[k]; ok {
+ t.Errorf("%s still sunk after UnrestAll", k)
+ }
+ }
+ if _, ok := sunk.m["officer/glm/glm-5"]; !ok {
+ t.Errorf("another key's sink was lifted")
+ }
+}
diff --git a/internal/gateway/routing.go b/internal/gateway/routing.go
index 2cf09f79f..7a4725cb9 100644
--- a/internal/gateway/routing.go
+++ b/internal/gateway/routing.go
@@ -687,7 +687,7 @@ func (s *Server) UnrestAll(key string) int {
}
liftedLocked(key, now)
restingUntil.Unlock()
- unsink(key)
+ unsinkAll(key)
s.trace.lifted()
return n
}
@@ -1114,7 +1114,15 @@ func (s *Server) restAfterMarked(c candidate, status int, header http.Header, bo
d, r.By, r.Failures = s.rateRest(c, header, body, sharedPool, now)
// rate limited with quota left: to the back, where the order sinks
if c.full(now).IsZero() && !(c.isOpenRouterFree() && sharedPool) {
- sink(c.restKey(), now)
+ // a remote magpie's rest is the model's alone (caa53673, the
+ // key's own limit apart): the sink keeps the same scope, or
+ // one model's 429 would send the remote's other models, never
+ // limited, behind the others too
+ if remoteModelRest(c, failRate, header) {
+ sink(c.restID(), now)
+ } else {
+ sink(c.restKey(), now)
+ }
}
case failRestricted:
// out until the vendor lifts it: an appeal, not a wait (#1849)
diff --git a/internal/gateway/sink.go b/internal/gateway/sink.go
index 5f1010b2b..e956e64c5 100644
--- a/internal/gateway/sink.go
+++ b/internal/gateway/sink.go
@@ -21,6 +21,7 @@ package gateway
import (
"slices"
+ "strings"
"sync"
"time"
@@ -55,10 +56,27 @@ func unsink(key string) {
sunk.Unlock()
}
-// sunkAt is when a candidate sank, if it did.
+// unsinkAll is unsink for key and each of its models (restID, key/model):
+// UnrestAll's lift of them all.
+func unsinkAll(key string) {
+ sunk.Lock()
+ for k := range sunk.m {
+ if k == key || strings.HasPrefix(k, key+"/") {
+ delete(sunk.m, k)
+ }
+ }
+ sunk.Unlock()
+}
+
+// sunkAt is when a candidate sank, if it did: its model alone, as
+// another magpie's model rate limited there sinks (remoteModelRest,
+// #2167), or its whole key or account.
func sunkAt(c candidate) (sinking, bool) {
sunk.Lock()
defer sunk.Unlock()
+ if s, ok := sunk.m[c.restID()]; ok {
+ return s, true
+ }
s, ok := sunk.m[c.restKey()]
return s, ok
}