Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

routing: Try now from the editor reaches the Routing page, and puts a sunk key back

yetone67361b54fb74
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index fa17f69e0..b311a563d 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -35,7 +35,7 @@ backfilled; see [Gateway Session Aggregation](gateway-sessions.md).
| Candidates | `plan` lists one provider's candidates: one per key or account on, then each `Fallback` model, with resting ones last (`restLast`). `planGroup` and `planLevel` weigh a group's members together. A nested group plans by its own routing (#576). So does a member whose provider routes other than Smart (`ownRouting`): weighed as one by its first not resting (`firstAwake`), tried whole in its own order; `leastHeld` leaves such a group's order alone too. A provider on Smart, or routed as the group is, is pooled with the group's other keys and accounts. `group_own_routing_test.go`. A member's own fallbacks are not the group's. | [`fallback.go`](../../internal/gateway/fallback.go) |
| Order of keys and accounts | `Routing`: smart (""), `order`, `rotate`, `usage` (least used, with an `usageHalfLife` of 1h), `pace`, `weight` or `free` (free first). `weighRouted` reads each candidate into a `routes.Seat` and asks the strategy serving its routing (`routes.Find`, the router strategy point of [the plugin runtime](plugin-runtime.md#gateway-extension-points-p3-router-strategy)) for the order: Smart is `routes.Smart`, the core's; the others are `routes.Builtin`, registered by the built-in plugin `rotation`. With it off a routing no strategy serves is tried in its listed order, and `routingOf` treats it as `order` wherever the core asks about a routing (a turn's mark, `goesRound`, `sinks`, a rotate group's sessions, a group's heads). Smart, `usage` and `pace` weigh a subscription by its allowance (`allowances`), and a key by its own usage windows when magpie reads them (`keyAllowance`: a sub2api key given a 5-hour, day or 7-day limit, or a key bought as a plan, such as Kimi Code's or a GLM Coding Plan's); any other key has none. `free` weighs by what a model costs there and nothing else: only a strategy with `routes.Strategy.ByFree` set (`routes.FreeFirst` is the one) has `weighRouted` fill `routes.Seat.Free`, from `provider.FreeModel` — the vendor's own `catalog.Model.Free` word, else a list price of $0 on every count — so a pool routed any other way never reads a model list for it, and one routed `free` spends the members whose model costs nothing before those that spend the allowance or credits, each tier in its listed order (nothing free is left behind a paid one, and a pool with no free model behaves as `order`). A group routed `free` weighs its members whole, each over its own accounts or keys as its provider routes them. A group routed `weight` (#1910) takes its members, not their keys, by smooth weighted round robin: `planLevel` makes each model member (its keys ordered by its own provider's routing) and each nested group one unit, and `weightedUnits` puts the round's pick (`routes.Round` over `group/<id>`, keyed by the member's spelling, `Group.Weights`, 1 if unset) first and the rest in member order for failover; a unit whose head rests leaves the round, while one whose accounts are only near their limit (Smart's 90% band, not resting) keeps its share, the band ordering only its own accounts (`TestWeightedGroupGivesALowGroupItsShare`; Creat.wu on Discord read the Routing page's "kept for when the others can't" as the member passed over, so the page now tells a member behind another by a group's weights or order as such, `memberSplit` in the routing plugin's `client.js`), and a kept conversation moves the round's charge to the member that kept it (`wrrSeat`, `routes.MoveRound`). A lone key or account is weighed against nobody, but what it has left is still read into the trace (`weighRouted`), so the Routing page shows it as it does for several (#1016); all but used up, it keeps its conversation rather than being said to move (`affine`). A failure rests a candidate (`failure`, `failureOf`). | [`routing.go`](../../internal/gateway/routing.go), [`weighted.go`](../../internal/gateway/weighted.go), [`internal/routes/`](../../internal/routes), [`internal/provider/routing.go`](../../internal/provider/routing.go) |
| Limits | Each key or account (a candidate's `who()`) has at most `MaxConcurrency` requests out at once (its own `AccountConcurrency` over it); one more waits in its lane's queue, bounded by `QueueLimit` and `QueueWait` and turned away with a 429 past either (`lanes.take`). `MaxRPM` (coeo91 on Discord: OpenRouter's free models take 20 a minute) bounds how many requests it sends the vendor in any rolling 60 seconds (`rpms`), counting each for a second more (`rpmSlack`), since the vendor counts a request when it arrives and the first on a new connection arrives later by its dial: one more waits for room in the minute, in the order they came, after it has its slot and right before it is sent. A wait that would be longer than 2 minutes (or the provider's `QueueWait`, when shorter: `rpmWait`) is turned away at once with a 429 whose `Retry-After` is when the minute has room; nothing rests for it. An agent gone while it waits gives its place back. Every request that reaches the vendor counts: each try of the candidate loop (a retry, the next key, a fallback) waits in `serve`, and the requests a try makes on the way and the gateway's own asks (count_tokens, System One's decisions, embeddings, drawing, Kimi's search) are counted by `rpmTransport` on `s.client` (`metered`, `paidFor`). count_tokens never waits: a candidate with no room is skipped, and the local estimate answers. A full lane or a minute with no room gives way to a ready key or account of the provider or the group (`laneMate`, `ready`); a fallback is never asked for want of either. A plugin's account sends through its own transport, so only its tries are counted, not a plugin's own extra requests. Set in the provider editor (Concurrency, Requests per minute, Queue size, Queue wait) or with `magpie provider account-concurrency`, `queue` and `rpm`; a subscription's are kept with its picks, and every account of it keeps them, those on beside the first as well (`withPicks`, called by `All` and `AlsoOn`; #1899: a second WorkBuddy AI account had none, and queued 62s with the wait set to 1s). | [`concurrency.go`](../../internal/gateway/concurrency.go), [`rpm.go`](../../internal/gateway/rpm.go), [`internal/provider/concurrency.go`](../../internal/provider/concurrency.go) |
-| Sink | With `Sink` on, a key or account that answers 429 while it still has quota goes to the back, behind every one not rate limited since. Held in memory until restart. A group in a group sinks by its own Sink, within its own order (`sinkOwn`, #2255), and the group asked for then sinks the whole by its own; `TestNestedGroupSinksItsOwnOrder`. | [`sink.go`](../../internal/gateway/sink.go) |
+| Sink | With `Sink` on, a key or account that answers 429 while it still has quota goes to the back, behind every one not rate limited since. Held in memory until restart, or until its rest is lifted by hand (Try now, `Unrest`/`UnrestAll` call `unsink`), which puts it back in its place (`TestTryNowPutsASunkAccountBack`). A group in a group sinks by its own Sink, within its own order (`sinkOwn`, #2255), and the group asked for then sinks the whole by its own; `TestNestedGroupSinksItsOwnOrder`. | [`sink.go`](../../internal/gateway/sink.go) |
| Image generation | `/v1/images/generations` and `/v1/images/edits` are translated to the selected model's image API. OpenRouter image-only models use its `/images` endpoint and send reference images as `input_references`; OpenRouter models that return images through chat stay on Chat Completions. Volcengine Ark (`isArk`: an `ark.*.volces.com` base, the preset's plans or one added by hand) is asked at `/images/generations` only, as its doc (https://www.volcengine.com/docs/82379/1541523) takes it (`drawArk`): `watermark: false`, since it marks every image otherwise; reference images as JSON `image` (a URL or data URL, an array for several), never a multipart edit; no `n`, so one request per image (`eachDrawing`); no ratio as `size`, so a ratio its doc lists is sent as that ratio's 2K size and any other is a 400 that says to put it in the prompt; no mask, no quality (JL on Discord, `TestArkDrawsWithoutWatermarkOncePerImage`). A routing group named for a drawing or a video (`drawingGroup`: `group/<id>`, or a bare id the group goes by that resolved to one of its members) is tried member by member in the order its routing plans for a chat (`drawingTries` over `planGroup`), each member over its own keys or accounts (`drawOnAccounts`, `startVideo`); the next member is asked while one fails as a chat's would move on (`retryable`: its vendor 5xx, out of quota, the model not served), never for a refusal of the request itself, and the call's `Fallback` names those tried first (#2005, tianxia3111: a group of two relays answered the first one's 502). Pause rules and a gateway key's group hold apply as for a chat. A failed drawing rests no member and no key. An account out of quota for the image model (`failure` reads `failQuota`) rests that model on it alone (`restDrawer`, keyed by `restID`), until the reset its refusal names (`resetsAt`: Antigravity's "Your quota will reset after 3h19m30s.") or `quotaRest`, so the next drawing asks the accounts that still have it first (`restLast`), and the account's text models stay in. Antigravity also tells a model's quota used up as a 200 whose stream is the error; `drawCodeAssist` gives such an error the status its words stand for (`saidStatus`: quota or rate 429, credit 402, else 502), as the images API's 200 with no image does, so a spent account hands the drawing to the next one as its 429 does (01huadalang on Discord: a 502 ended the drawing on the spent account; `TestAntigravityDrawingOutOfQuotaInsideTheReplyGoesOn`, `TestAntigravityDrawingQuotaRestsTheImageModel`). `TestImageGroupMovesOnToTheNextMember`, `TestVideoGroupMovesOnToTheNextMember`. An account's own list of models (#474) never bars a ChatGPT account's GPT Image, a Grok subscription's Grok Imagine or a WorkBuddy plan's image models (`listsCantName`): the list is picked from the editor's chat models, which they are never among, so a list kept for some chat models had every drawing refused as "set not to be served by any account" (heyiwei5 on X). A key's list, picked from models that include its image ones, still bars them. `TestCodexDrawsPastAccountModelLists`. A drawing is given Settings → Images → Image generation timeout (`imageTimeout`, 1 to 60 minutes, 5 when unset; neighbads on X: a slow image endpoint needed more) as its context's deadline (`drawTimeout`, `drawWithin`), and so is downloading an image answered as a link (`imageLongest`, never under `ImageLongest`). A drawing's request goes through a transport with no `ResponseHeaderTimeout` (`drawsApart`), since an images API sends its headers only once it has drawn, and every other request keeps `upstreamHeaderWait` (10 minutes). A video's submit, status and download calls keep 5 minutes each (`videoTimeout`). The `magpie mcp image` process, which may not read these settings (WSL), waits on a call up to `callLongest` (two hours and a minute), so it never cuts one the gateway still waits on. `TestLongImageTimeoutLetsASlowVendorFinish`, `TestImageTimeoutSettingReachesTheVendor`, `TestOnlyDrawingsWaitPastTheHeaderWait`. | [`draw.go`](../../internal/gateway/draw.go), [`draw_volcengine.go`](../../internal/gateway/draw_volcengine.go), [`video.go`](../../internal/gateway/video.go), [`catalog.go`](../../internal/catalog/catalog.go) |
| Affinity | A conversation stays with the key or account that answered it, so the vendor's prompt cache is read again. Modes: auto (""), `session`, `turn`, `off`, `strict`. Stored in `affinity.json` next to `providers.json`. | [`affinity.go`](../../internal/gateway/affinity.go) |
| One try | `attempt` sends one candidate the request. It goes through the proxy of the account or key it uses, else the provider's (`p.Via`, `ProxyChoice`: `AccountProxies` holds an account's by its name, a key's by its `KeyID`). A request magpie makes for one provider while serving another's (a web search, an image description, a classifier call) takes its own provider's proxy or the global one, never the outer one: `netproxy.With` replaces a choice the context already names, "" too, and `webSearch` starts from none, so the search APIs follow the global proxy (#1522). A provider of the user's set not to check its certificate (`InsecureTLS`, #2383) has it skipped for its own base URLs' hosts alone, through `p.Via` ([Providers and accounts](providers-accounts.md)). Every request to one vendor's host shares one HTTP/2 connection (`upstreamTransport`); one with no frame for 20s is pinged, and dropped when no answer comes within 10s (`netproxy.HTTP2Health`, on `http.DefaultTransport` too), so a connection gone silent without closing (a proxy's way out died, a NAT forgot the flow) fails the requests on it, which move on as a 502 does, and the next request dials a new one. A live vendor answers a ping whatever its streams are doing, so a long wait for a first byte is not cut by it (#2843: a group's requests hung with no first byte until a restart; `TestGroupMovesOnFromSilentHTTP2Connection`). A subscription with its own client is served specially, such as Claude through the genuine Claude Code. The request is relayed as is (`passthrough`) when the model is usable on the agent's API, except that a Claude model where the provider has an Anthropic URL is translated to Messages from Chat or Responses too (`OnMessages`: OpenCode speaks Chat for every model, and a relay drops cache_control there), unless the user set the model's API or the vendor's list names its APIs. An account that only streams (`Account.Stream`: a plugin's, and most subscriptions magpie signs in to) has a request that asked for one body translated, which magpie always asks streamed, except a plugin's on Responses (`gathers`): that is relayed all the same, asked with `stream:true`, and the client given the `response` of the stream's `response.completed` (or `response.incomplete`) as one JSON body (`gathered` in [`gathered.go`](../../internal/gateway/gathered.go); its `output` filled from the stream's `response.output_item.done` items, or else its text, when that event's is empty). So the tools the vendor runs itself, and its response's `tools` and usage details, reach the client (plugins#112: a translation dropped Grok's `x_search`, and Grok answered without searching; `TestGrokPluginNonStreamKeepsHostedTools`). A stream that fails or ends before that event is a 502 (`TestGrokPluginNonStreamEndedShort`), never an empty 200. A Messages request is still translated, its stream having no event that holds the whole reply, and so is a subscription magpie signs in to itself (the ChatGPT backend's, Qoder's): its own code fits its requests. A request offering web search to a provider that doesn't search by itself is translated too, so magpie can search for it (`search.go`; a provider that searches for some of its models only, Command Code's Provider API, lists them in `searchModels` by API, and none count under its `x-cmd-zdr` header, #1590), but on the agent's own API where the provider serves it and no `ResponsesFirst`/`MessagesFirst` rule prefers another: Codex offers web_search on every turn, and each went out as Chat (#997). Otherwise it is translated (`translate`) to the first API `usable` gives: Chat, except Responses for OpenAI's models where their maker serves them (`ResponsesFirst`), Responses for every model at ModelScope, whose Chat API reports neither the tokens its cache answered nor reasoning while its Responses API reports both (`ResponsesTellsCache`, #1578; a 404 there sends the model to Chat, and only there), and Anthropic's Messages for a Claude model where the provider has an Anthropic URL (`MessagesFirst`, which keeps its cache breakpoints, #997). An API that answers the model is not served there (`wrongEndpoint`) is remembered in `unfit` until magpie restarts. A Chat API that refuses function tools with reasoning, with no other API to take them to, is asked once more with `reasoning_effort` `none` (#176), unless `none` is known refused (`turnedAway`); `none` refused naming the levels, the tools refusal is the answer, not a retry at the lowest level (#2670). When every API has turned the model away, the API the user picked for it (`ModelAPI`) is asked again, and without a pick each API the provider has a URL for (`Speaks`), Chat first. `Provider.Native`, the API agents with a per-model API (Pi, omp, droid) are wired to ask on, follows the same order. A custom provider with a Gemini URL (#1346) has Gemini last among the APIs it speaks: a Gemini client's request is relayed there, and any other is translated to it when it has no other, or when the user set Gemini as the model's API (`SetModelAPI` takes chat, responses, anthropic or gemini, each only where the provider has its URL; a model set to Gemini has no `Native`, so those agents ask it on their own API and it is translated). Upstream it is always asked `models/{model}:streamGenerateContent?alt=sse` under the base (`upstreamPath`), the model only in the path (`forwardOnce` drops the body's), so a Gemini client that asked for one JSON body is translated, as for Factory's generate route. | [`gateway.go`](../../internal/gateway/gateway.go) |
@@ -116,7 +116,7 @@ recorded as a prediction failure (`noPrediction`).
- A rule by agent (`Rule.Agents`, `agents=codex,sillytavern`) matches the agent the gateway names the request's (`agentOf` in [`gateway.go`](../../internal/gateway/gateway.go): its `magpie-<name>` key, else its User-Agent), in lower case (`cleanRules`, `MatchesBesidesIntent`, `Pauses`). A client magpie doesn't know is named by its User-Agent's product name, before the first `/` or space (`usage.AgentOf`), so a rule can hold SillyTavern or one's own app (raikyou on Discord). The editor's agent picker lists the agents magpie knows, then those `/api/usage?period=30d` heard from that it doesn't (not `other`, nor `magpie`), and takes a name typed in its filter as the gateway reads a User-Agent (`uaName` in [`plugins/routing/client.js`](../../internal/gui/assets/plugins/routing/client.js), through the picker's `field.typed`): "MyApp/2.0 (Linux)" is `myapp`. `routing-rule-agent-typed.test.cjs`.
- A pause rule (`Rule.Pause`, `pause=<model>`; John on Discord: a model of the group paused in its vendor's peak hours) leaves the model it names out of the group while it holds (`provider.PausedOut`, on `ruleClock`): the group's own rules and each nested group's on the way down to a member, a group named taking all its models, and a model named taking it wherever a group in the group has it (#625). It runs before planning, on every request rather than once a turn, so the model is not first, not a failover, and not kept by affinity, and comes back as soon as the hours end. It holds by hours, days, agents and folders only (`cleanRules` refuses another condition), and is not a send-to rule: `Matches`, `MatchRule`, `ThenUses`, `Ruled` and the compaction rules leave it out. A send-to rule naming a paused model finds it unready and the group routes as without it. The trace keeps what was paused (`GroupRef.Paused`), and the Routing page says so. Every model paused answers 503 (`pausedError`) naming each rule. Embeddings go by the same pauses; manual groups have no rules (`Live`), so none pause there. A pause by folder (#1606, Ti800: a subscription's quota kept for the long task in one folder, `pause=<model> except-folders=~/big-task`) reads the folder only when such a pause is on the way (`provider.PausesByFolder`, then `workDir`): outside its except-folders, which includes a request that says no folder (a subagent's or a small model's that doesn't), the model is left out; with `folders=`, only requests known to be in them. Embeddings and System One's decision calls have no folder. `Paused.Folder`/`ByFolder` carry what was read, and the Routing page says the folder or that none was said. `rules_pause_test.go`, `TestPauseByFolder`, `TestPauseByFolderRule`, `TestParsePauseByFolder`, `routing-pause-folder.test.cjs`.
3. Candidates come from `planGroup` for a group or `plan` for a provider. They are filtered by `sealedReaders` and `allowedCandidates`. An account past its share is held out of the candidates (`capHeld`): a usage cap's share, each window at its own where the account gives it one (`AccountWindowCaps`, willz on Discord: 50% of the five hours, 40% of the week) and at the account's cap otherwise, or 100% of a window for a Codex account set not to spend its credits (`provider.HoldCaps`, `WindowCaps.Of`, `CodexNoCredits`). A ChatGPT account's call on `codex-auto-review`, which OpenAI counts in no window, is held by no usage cap, only by credits it may not spend (`freeOnChatGPT`, #1593, `TestAccountCapLetsAutoReviewThrough`). The account is held while any window is past its own share, and `cappedError` and the trace name the fullest such window's share and cap. When every candidate is held and one held for its credits spends its resets by itself, `autoReset` spends a reset on it and it is tried (`planned.held`); otherwise a capped or barred set returns `cappedError` or `barredError`. Then `affine` moves the conversation's last answerer first, unless it is resting or nearly used up. An `X-Magpie-Account` header (`AccountHeader`) pins the request to one account: only that account is tried, and when it can't answer the caller is told why.
-4. The candidate loop calls `attempt` on each candidate in turn, through a `holdWriter`. The writer holds the reply until it is clearly passing. A candidate that fails before anything is sent rests (`failure`): out of credit for 30 minutes, out of quota until reset, rate limited until retry-after, otherwise a minute and longer on repeats: doubled each time, up to `longestRetry` (10 minutes) for a failure and `longestRateRest` (30 minutes) for a rate limit back as soon as its rest is over, or up to the provider's own `restCap` in minutes (`Provider.RestCapOr`, at most `MaxQuotaRest`; `magpie provider rest-cap <id> 4h`, #2008: a free key down for hours was asked every 10 minutes). A time the vendor names stands over it as before (`TestOwnRestCap`). The failure count is kept per key or account (`restKey`), and any of its models' answers clears it (`served`). Each model's own count (`restID`) is kept beside it and cleared only by that model's answer (`servedCandidate`): once it is past the account's, a sibling model answered since it last did, so the failure is the model's and it climbs its own ladder and rests alone, its siblings still served (#2241, weizsw: an Antigravity seat's gemini-3.5-flash-lite rested a minute 41 times of 43 while gemini-3.8-flash kept answering; `TestModelLadderClimbsPastSiblingAnswers`). Every model failing with no answer between still climbs the account's ladder and rests the account, and a candidate naming no model has the account's count alone. `rest-cap <id> off` (`provider.RestCapOff`, `restCap: -1`) is for a relay or account pool whose unknown errors come now and then and pass by the next request (Kismet Alan on Discord): a failure that says nothing of why (`failOther`: a 5xx, a dropped connection) then rests nobody and isn't counted, its trace rest `by: "off"` with no time left. A 401 or 403, a model not served, an account's window full, a rate limit (magpie's own 30-minute cap), quota, credit and a restriction still rest it as before (`TestRestCapOff`). The shortest cap otherwise is a minute, the first rest; `rest-cap <id> 30s` is turned away and names `off`. A 400, 403, 404 or 422 that says the model isn't served (`unservedWords`) and says nothing of quota, credit or a rate limit rests only that key's or account's model (`modelRefused`, keyed by `restID`, its backoff counted apart): its other models are still served first in order (#1235, SenseNova's 403 "model is not available in the current token plan"). A Google account's Code Assist (the Antigravity and Gemini CLI built-ins) answering 404 `NOT_FOUND` ("Requested entity was not found.") rests its model the same way, as its endpoint is the same for every model (`codeAssistModelMissing`; Packing1 on Discord: Argon, listed for a Google AI Pro account, answered 404 and rested the whole account; `TestCodeAssistMissingModelRestsTheModel`). A refusal for want of quota, credit or rate, for an account its vendor restricted, or for a verify it asks of its account there, from another computer's magpie (a Remote magpie provider, LAN share) rests only the model it was about (`remoteModelRest`, keyed by `restID`, in `restAfterMarked` and `rateRest`): that magpie serves each of its models from its own providers and rests them itself, so one out of quota there says nothing of the others (bigsteven on Discord: a 429 for one model rested every model on the remote for the 3 hours it named, so a group never moved on to the remote's other models; `TestRemoteMagpieQuotaRestsTheModel`, `TestRemoteMagpieRestrictedRestsTheModel`). Its key refused, the computer unreachable, or the key's own limit there (`keyLimited`'s 429, which carries `X-Magpie-Limit-Reset`, `remoteKeyLimitHeader`), still rests it whole: #2077's reporter had a Claude Code session limit on the other magpie refuse its Qoder models here (`TestRemoteMagpieRestsTheModel`, two gateways over HTTP; `TestRemoteMagpieKeyLimitRestsTheKey`). A 400, 404, 410 or 422 that says the model is retired (`retiredWords`: OpenCode Zen's 410 `ModelDeprecated` "Model exo-free has been deprecated.", whose own model list still names it; MOMO on Discord) rests only that model the same way, is retryable so a group's next member or another provider of it is asked, and is remembered (`provider.Retire`, `retired.json` beside `providers.json`): the model leaves that provider's entries in the catalog (`buildEntries`), so /v1/models, every agent's model list and the found groups go without it, until a request on it is answered again (`Unretire` in `servedCandidate`) or a week has passed. A refresh of the vendor's list doesn't bring it back early, since that list is where it still is. A `provider/model` request for it is still sent. A 400 or 422 that says this upstream's API can't take the request's shape (`shapeWords`: xAI's 422 "unknown item type" (#350), OpenAI's "Unknown parameter", and a limit of a model's own on tool schemas, OpenCode Go's 400 "JSON schema exceeds the maximum nesting depth of 10 levels" on muse-spark-1.3-contributor, #1870), or that it is served on another API than the one asked (`wrongEndpoint`, once the member has tried every API it has a URL for: #2670, a relay with a Chat URL alone refusing tools with reasoning and `none` too), rests nobody: the next candidate is asked once (`shapeRefused`, `failShape`), and the last one's error reaches the agent. A request missing what every API requires ("field required") isn't one, nor is a 200 that mentions it. Command Code's Provider API turning a request away with nothing but a trace id (`unsaidWords`: `{"message":"invalid request error trace_id: <hex>","type":"invalid_request_error"}`, a 400, or a 422 before) is handled the same way: it turns away a request whose history and asked-for output together are over a limit below the model's listed context (the reporter's A/B: one ~748K-token agent conversation answered 200 with `max_completion_tokens` 64000 and this 400 with 384000, the model's listed max output), which another member's model may take, and the key is fine (#2313, `TestGroupFailsOverCommandCodesBare400`). Its overflow, which names the model's context length before the trace id, and an error that names a `param`, aren't one. A relay's own cap on one request's input that says so (`tooLongRe`: B.ai's 400 "Input token exceed the limit", on a 380K request under the model's 1M window; #2445, worksyang) is taken as an overflow (`overflowed`), which no word list of `retryable` matched: a group's next member that may hold the request is asked (`withRoom`, `failOverflow`), nobody rests, another account of the same model isn't asked, and with no member left the agent is told the conversation is too long, so it compacts. A rate limit worded with input tokens (`paceWords`) isn't one, and a plain bad request is still asked of nobody else (`TestRelayInputCapGoesToTheNextMember`). A subscription account that fails any other way with a window full, by magpie's latest reading of its windows, rests until that window renews. A 401 or 403 is the exception: one that names a quota is out of quota already (`usedUpWords`), and one that doesn't is about the key or the request, so it rests on the backoff ladder whatever the reading says (plugins#243, zygs1083-dotcom: a Factory account read at 98% sat out its week over one plain 403; `TestPlain403DoesNotRestUntilTheWindow`). A plain 403 (`plainForbidden`: no quota, credit, rate, verification, restriction or model not served) from a second key or account of the same provider, model and reasoning in one request is the request refused, not each account: neither rests, the first one's rest and counted failure are taken back (`forbiddenAlike`, `unforbid`), and the provider's other mates are asked last (`matesLast`). Before, that one request walked every account of the provider into backoff (`TestPlain403OnTwoMatesRestsNeither`). One account's plain 403 alone still rests it. Both quota rests and window rests are held to `longestQuota` (8 days), so a reset time read wrong (a year off) doesn't bench the account for that long. The next candidate is then tried. A failed provider waits `fallbackCooldown` (1 minute) at the back. A WorkBuddy account the vendor restricted (#1849: 403 `{"code":11140,"msg":"request illegal","displayMsg":{"en":"Your account has been restricted due to unusual activity",…}}`) rests `restrictedRest` (2 hours) as `restricted` (`failRestricted`), not the minutes of an unknown failure: only WorkBuddy lifting the restriction, after the user's appeal, brings it back. Both shapes are known: the built-in's `wbExplain` and the plugin's `explained()` (0.1.17, the mover's `min`) end the error with `provider.WBRestrictedHint` (`restrictedSaid`), and the vendor's own body, its code 11140 or its words (`provider.WBRestricted`), counts only from a WorkBuddy candidate (`restrictedRefused`, `isWorkBuddy`), as another vendor's 11140 is its own. Bare "request illegal" isn't matched. WorkBuddy's 11128 channel refusal stays `promptRefused` and rests nobody. The Routing page names it "account restricted by the vendor", says the appeal is the way back, and gives the account's row a button that lifts the rest (`/api/gateway/unrest`) once the vendor has. Every other rest a key or account takes (quota, rate limit, retry-after, a failure's backoff) can be lifted by hand the same way (#1869): "Try now" on its row on the Routing page and beside its rest note in the provider's key list. It ends the rest only: the backoff count stays, and a refusal after it rests the key again the usual way. The vendor's safety filter refusing a reply rests nobody, whether it refuses the turn before any of the reply is said or once some of it has been said (`refused`, `refusedAfter`). So does one said in a stream a client is answered whole from (a Codex account is streamed upstream for a `stream:false` Responses, Chat or Messages request): each decoder gives the refusal's error event status 400 and its code (`filterRefused`), a whole reply it cut is answered the refusal rather than its words so far, and Anthropic's error shape, which has no code, carries it in `refusedHeader`; before, it was answered 502 `api_error`, asked again, and each account rested (#3274). `restAfterMarked` itself rests nobody, and counts no failure, for a 400 whose error object `policyRefusal` reads as a refusal, by its code, never by words a body may echo of the request. Its note on the try (and in the error the agent gets when nobody is left) is the vendor's reason, with Anthropic's `stop_details` category and explanation when it gave them (`refusedDetail`, #1768: "safety filter: refusal (category cyber)"); none is made up when it gave none. `stop_details` is carried through the IR (`Event.StopDetails`), so a Claude subscription run through the claude CLI and a translated Anthropic route keep it for an Anthropic client too. How many more candidates a refusal before any of the reply is asked of is Settings' `RefusalRetries` (`refusalAsksAgain`, counted per request in the candidate loop; a group's other members count too): unset, every one left (#248); 0, none (#1925, DD on Discord). When nobody is left, an agent on Anthropic's Messages API gets the held refusal itself, the vendor's 200 with `stop_reason` `refusal` and its `stop_details`, streamed or whole (`holdWriter.handBack`), since that is what Claude Code switches to its fallback model on; a 400 is only an API error to it. Codex and Chat clients still get `failTo`'s 400 (`refusedStatus`), which they don't ask again for. `TestRefusalRetries*` in [`refusal_retries_test.go`](../../internal/gateway/refusal_retries_test.go) cover keys, a group and the Claude subscription. The ChatGPT backend's 502 "response protection is unavailable" is about the request's content, not the account: the same history fails the same on each of its accounts and models. It rests nobody, isn't tried again after a moment, and isn't asked of the provider's other accounts or models; another provider left in a group is asked once (`protectionRefused`, `elsewhere`). One cause is known (#1270): a history that replays the backend's own `web_search_call` in a request that doesn't declare the `web_search` tool is refused so, whatever the account, model or turn (a plain turn, a thread's description, a compaction, Codex's own text compaction for a custom provider with `tools: []`). Every request to the ChatGPT backend with such a history therefore declares cached-only search (`{"type":"web_search","external_web_access":false}`, `provider.DeclareSearch`): an account's in `codexBody`, Codex's own sign-in relayed in `codexUpstreamOn`. It goes in the tools, or for Responses Lite (its header, `reasoning.context` `all_turns`, or an `additional_tools` item already there), which refuses a top-level `web_search`, in the `additional_tools` item. A request that offered no tools gets `tool_choice` `none` so the model can't search; one with a forced choice keeps it. A history without a search, or a request that declares search already, is relayed unchanged; Codex's own `web.run` (a function in the `web` namespace, which Codex offers in place of the hosted tool on a Responses Lite model or with its standalone web search on, and runs through `/alpha/search`) counts as declaring it, since the backend refuses a request carrying both `web.run` and the hosted `web_search` (Savin on Discord, Codex 0.163.0-alpha; `TestSearchedConversationUnderWebRun`). siwc, API-key requests to openai.com and the legacy `/responses/compact` are left as they were. Command Code turning a model away for its Zero Data Retention setting (its plugin's switch, 0.1.14, which sends `x-cmd-zdr: 1`; magpie-community/plugins#246) is a refusal of that request's setting, not of the model or the account: the Provider API's 422 with `"code":"cmd_zdr_no_providers"` and the 400 the plugin makes of Go's error line with `"type":"cmd_zdr_no_providers"` (`zdrCoded`, matched as the error's code or type, never by its words, whose "…a different model.","type":"invalid_request_error" read as a model unserved and rested the model on the account). `keepRetry` notes it beside the error (`zdrHeader`), since the error the agent gets may keep no code (Anthropic's shape, or the 400's numeric code). It rests nobody, anywhere (`restAfterMarked` returns `by: "off"`), so the model answers as soon as the switch is off; it isn't asked of the provider's other accounts at that model, which carry the same header (`notMates`), nor again after a moment; another member, the provider's other models among them, is asked once, and with none left the agent gets Command Code's words (`TestCommandCodeZDRRefusalRestsNothing`, `TestZDRRefusalNeverRests`). Another vendor's refusal in the same words, without the code, still rests the model (`TestZDRRefusalIsCommandCodesCodeOnly`).
+4. The candidate loop calls `attempt` on each candidate in turn, through a `holdWriter`. The writer holds the reply until it is clearly passing. A candidate that fails before anything is sent rests (`failure`): out of credit for 30 minutes, out of quota until reset, rate limited until retry-after, otherwise a minute and longer on repeats: doubled each time, up to `longestRetry` (10 minutes) for a failure and `longestRateRest` (30 minutes) for a rate limit back as soon as its rest is over, or up to the provider's own `restCap` in minutes (`Provider.RestCapOr`, at most `MaxQuotaRest`; `magpie provider rest-cap <id> 4h`, #2008: a free key down for hours was asked every 10 minutes). A time the vendor names stands over it as before (`TestOwnRestCap`). The failure count is kept per key or account (`restKey`), and any of its models' answers clears it (`served`). Each model's own count (`restID`) is kept beside it and cleared only by that model's answer (`servedCandidate`): once it is past the account's, a sibling model answered since it last did, so the failure is the model's and it climbs its own ladder and rests alone, its siblings still served (#2241, weizsw: an Antigravity seat's gemini-3.5-flash-lite rested a minute 41 times of 43 while gemini-3.8-flash kept answering; `TestModelLadderClimbsPastSiblingAnswers`). Every model failing with no answer between still climbs the account's ladder and rests the account, and a candidate naming no model has the account's count alone. `rest-cap <id> off` (`provider.RestCapOff`, `restCap: -1`) is for a relay or account pool whose unknown errors come now and then and pass by the next request (Kismet Alan on Discord): a failure that says nothing of why (`failOther`: a 5xx, a dropped connection) then rests nobody and isn't counted, its trace rest `by: "off"` with no time left. A 401 or 403, a model not served, an account's window full, a rate limit (magpie's own 30-minute cap), quota, credit and a restriction still rest it as before (`TestRestCapOff`). The shortest cap otherwise is a minute, the first rest; `rest-cap <id> 30s` is turned away and names `off`. A 400, 403, 404 or 422 that says the model isn't served (`unservedWords`) and says nothing of quota, credit or a rate limit rests only that key's or account's model (`modelRefused`, keyed by `restID`, its backoff counted apart): its other models are still served first in order (#1235, SenseNova's 403 "model is not available in the current token plan"). A Google account's Code Assist (the Antigravity and Gemini CLI built-ins) answering 404 `NOT_FOUND` ("Requested entity was not found.") rests its model the same way, as its endpoint is the same for every model (`codeAssistModelMissing`; Packing1 on Discord: Argon, listed for a Google AI Pro account, answered 404 and rested the whole account; `TestCodeAssistMissingModelRestsTheModel`). A refusal for want of quota, credit or rate, for an account its vendor restricted, or for a verify it asks of its account there, from another computer's magpie (a Remote magpie provider, LAN share) rests only the model it was about (`remoteModelRest`, keyed by `restID`, in `restAfterMarked` and `rateRest`): that magpie serves each of its models from its own providers and rests them itself, so one out of quota there says nothing of the others (bigsteven on Discord: a 429 for one model rested every model on the remote for the 3 hours it named, so a group never moved on to the remote's other models; `TestRemoteMagpieQuotaRestsTheModel`, `TestRemoteMagpieRestrictedRestsTheModel`). Its key refused, the computer unreachable, or the key's own limit there (`keyLimited`'s 429, which carries `X-Magpie-Limit-Reset`, `remoteKeyLimitHeader`), still rests it whole: #2077's reporter had a Claude Code session limit on the other magpie refuse its Qoder models here (`TestRemoteMagpieRestsTheModel`, two gateways over HTTP; `TestRemoteMagpieKeyLimitRestsTheKey`). A 400, 404, 410 or 422 that says the model is retired (`retiredWords`: OpenCode Zen's 410 `ModelDeprecated` "Model exo-free has been deprecated.", whose own model list still names it; MOMO on Discord) rests only that model the same way, is retryable so a group's next member or another provider of it is asked, and is remembered (`provider.Retire`, `retired.json` beside `providers.json`): the model leaves that provider's entries in the catalog (`buildEntries`), so /v1/models, every agent's model list and the found groups go without it, until a request on it is answered again (`Unretire` in `servedCandidate`) or a week has passed. A refresh of the vendor's list doesn't bring it back early, since that list is where it still is. A `provider/model` request for it is still sent. A 400 or 422 that says this upstream's API can't take the request's shape (`shapeWords`: xAI's 422 "unknown item type" (#350), OpenAI's "Unknown parameter", and a limit of a model's own on tool schemas, OpenCode Go's 400 "JSON schema exceeds the maximum nesting depth of 10 levels" on muse-spark-1.3-contributor, #1870), or that it is served on another API than the one asked (`wrongEndpoint`, once the member has tried every API it has a URL for: #2670, a relay with a Chat URL alone refusing tools with reasoning and `none` too), rests nobody: the next candidate is asked once (`shapeRefused`, `failShape`), and the last one's error reaches the agent. A request missing what every API requires ("field required") isn't one, nor is a 200 that mentions it. Command Code's Provider API turning a request away with nothing but a trace id (`unsaidWords`: `{"message":"invalid request error trace_id: <hex>","type":"invalid_request_error"}`, a 400, or a 422 before) is handled the same way: it turns away a request whose history and asked-for output together are over a limit below the model's listed context (the reporter's A/B: one ~748K-token agent conversation answered 200 with `max_completion_tokens` 64000 and this 400 with 384000, the model's listed max output), which another member's model may take, and the key is fine (#2313, `TestGroupFailsOverCommandCodesBare400`). Its overflow, which names the model's context length before the trace id, and an error that names a `param`, aren't one. A relay's own cap on one request's input that says so (`tooLongRe`: B.ai's 400 "Input token exceed the limit", on a 380K request under the model's 1M window; #2445, worksyang) is taken as an overflow (`overflowed`), which no word list of `retryable` matched: a group's next member that may hold the request is asked (`withRoom`, `failOverflow`), nobody rests, another account of the same model isn't asked, and with no member left the agent is told the conversation is too long, so it compacts. A rate limit worded with input tokens (`paceWords`) isn't one, and a plain bad request is still asked of nobody else (`TestRelayInputCapGoesToTheNextMember`). A subscription account that fails any other way with a window full, by magpie's latest reading of its windows, rests until that window renews. A 401 or 403 is the exception: one that names a quota is out of quota already (`usedUpWords`), and one that doesn't is about the key or the request, so it rests on the backoff ladder whatever the reading says (plugins#243, zygs1083-dotcom: a Factory account read at 98% sat out its week over one plain 403; `TestPlain403DoesNotRestUntilTheWindow`). A plain 403 (`plainForbidden`: no quota, credit, rate, verification, restriction or model not served) from a second key or account of the same provider, model and reasoning in one request is the request refused, not each account: neither rests, the first one's rest and counted failure are taken back (`forbiddenAlike`, `unforbid`), and the provider's other mates are asked last (`matesLast`). Before, that one request walked every account of the provider into backoff (`TestPlain403OnTwoMatesRestsNeither`). One account's plain 403 alone still rests it. Both quota rests and window rests are held to `longestQuota` (8 days), so a reset time read wrong (a year off) doesn't bench the account for that long. The next candidate is then tried. A failed provider waits `fallbackCooldown` (1 minute) at the back. A WorkBuddy account the vendor restricted (#1849: 403 `{"code":11140,"msg":"request illegal","displayMsg":{"en":"Your account has been restricted due to unusual activity",…}}`) rests `restrictedRest` (2 hours) as `restricted` (`failRestricted`), not the minutes of an unknown failure: only WorkBuddy lifting the restriction, after the user's appeal, brings it back. Both shapes are known: the built-in's `wbExplain` and the plugin's `explained()` (0.1.17, the mover's `min`) end the error with `provider.WBRestrictedHint` (`restrictedSaid`), and the vendor's own body, its code 11140 or its words (`provider.WBRestricted`), counts only from a WorkBuddy candidate (`restrictedRefused`, `isWorkBuddy`), as another vendor's 11140 is its own. Bare "request illegal" isn't matched. WorkBuddy's 11128 channel refusal stays `promptRefused` and rests nobody. The Routing page names it "account restricted by the vendor", says the appeal is the way back, and gives the account's row a button that lifts the rest (`/api/gateway/unrest`) once the vendor has. Every other rest a key or account takes (quota, rate limit, retry-after, a failure's backoff) can be lifted by hand the same way (#1869): "Try now" on its row on the Routing page and beside its rest note in the provider's key list. It ends the rest, and with Sink on puts the key back from the back of the order; the backoff count stays, and a refusal after it rests the key again the usual way. Every lift before its time (Try now, `UnrestAll` for a replaced key, `renewed` for windows that renewed) is noted by its rest key (`restingUntil.lifted`) and handed to the Routing page in the trace (`TraceState.Lifted`, `lifted` in `/api/gateway/trace`), and a lift wakes a page waiting on the trace though no route changed (`trace.lifted`). The page stops showing a rest or a sink found before its lift, wherever the lift was asked for (zoecats on Discord: Try now in the provider's editor left Routing at 429 until a restart; `TestTryNowToldToTheRoutingPage`, `routing-lifted-elsewhere.test.cjs`). With another magpie serving the gateway, Try now lifts nothing (#3555). The vendor's safety filter refusing a reply rests nobody, whether it refuses the turn before any of the reply is said or once some of it has been said (`refused`, `refusedAfter`). So does one said in a stream a client is answered whole from (a Codex account is streamed upstream for a `stream:false` Responses, Chat or Messages request): each decoder gives the refusal's error event status 400 and its code (`filterRefused`), a whole reply it cut is answered the refusal rather than its words so far, and Anthropic's error shape, which has no code, carries it in `refusedHeader`; before, it was answered 502 `api_error`, asked again, and each account rested (#3274). `restAfterMarked` itself rests nobody, and counts no failure, for a 400 whose error object `policyRefusal` reads as a refusal, by its code, never by words a body may echo of the request. Its note on the try (and in the error the agent gets when nobody is left) is the vendor's reason, with Anthropic's `stop_details` category and explanation when it gave them (`refusedDetail`, #1768: "safety filter: refusal (category cyber)"); none is made up when it gave none. `stop_details` is carried through the IR (`Event.StopDetails`), so a Claude subscription run through the claude CLI and a translated Anthropic route keep it for an Anthropic client too. How many more candidates a refusal before any of the reply is asked of is Settings' `RefusalRetries` (`refusalAsksAgain`, counted per request in the candidate loop; a group's other members count too): unset, every one left (#248); 0, none (#1925, DD on Discord). When nobody is left, an agent on Anthropic's Messages API gets the held refusal itself, the vendor's 200 with `stop_reason` `refusal` and its `stop_details`, streamed or whole (`holdWriter.handBack`), since that is what Claude Code switches to its fallback model on; a 400 is only an API error to it. Codex and Chat clients still get `failTo`'s 400 (`refusedStatus`), which they don't ask again for. `TestRefusalRetries*` in [`refusal_retries_test.go`](../../internal/gateway/refusal_retries_test.go) cover keys, a group and the Claude subscription. The ChatGPT backend's 502 "response protection is unavailable" is about the request's content, not the account: the same history fails the same on each of its accounts and models. It rests nobody, isn't tried again after a moment, and isn't asked of the provider's other accounts or models; another provider left in a group is asked once (`protectionRefused`, `elsewhere`). One cause is known (#1270): a history that replays the backend's own `web_search_call` in a request that doesn't declare the `web_search` tool is refused so, whatever the account, model or turn (a plain turn, a thread's description, a compaction, Codex's own text compaction for a custom provider with `tools: []`). Every request to the ChatGPT backend with such a history therefore declares cached-only search (`{"type":"web_search","external_web_access":false}`, `provider.DeclareSearch`): an account's in `codexBody`, Codex's own sign-in relayed in `codexUpstreamOn`. It goes in the tools, or for Responses Lite (its header, `reasoning.context` `all_turns`, or an `additional_tools` item already there), which refuses a top-level `web_search`, in the `additional_tools` item. A request that offered no tools gets `tool_choice` `none` so the model can't search; one with a forced choice keeps it. A history without a search, or a request that declares search already, is relayed unchanged; Codex's own `web.run` (a function in the `web` namespace, which Codex offers in place of the hosted tool on a Responses Lite model or with its standalone web search on, and runs through `/alpha/search`) counts as declaring it, since the backend refuses a request carrying both `web.run` and the hosted `web_search` (Savin on Discord, Codex 0.163.0-alpha; `TestSearchedConversationUnderWebRun`). siwc, API-key requests to openai.com and the legacy `/responses/compact` are left as they were. Command Code turning a model away for its Zero Data Retention setting (its plugin's switch, 0.1.14, which sends `x-cmd-zdr: 1`; magpie-community/plugins#246) is a refusal of that request's setting, not of the model or the account: the Provider API's 422 with `"code":"cmd_zdr_no_providers"` and the 400 the plugin makes of Go's error line with `"type":"cmd_zdr_no_providers"` (`zdrCoded`, matched as the error's code or type, never by its words, whose "…a different model.","type":"invalid_request_error" read as a model unserved and rested the model on the account). `keepRetry` notes it beside the error (`zdrHeader`), since the error the agent gets may keep no code (Anthropic's shape, or the 400's numeric code). It rests nobody, anywhere (`restAfterMarked` returns `by: "off"`), so the model answers as soon as the switch is off; it isn't asked of the provider's other accounts at that model, which carry the same header (`notMates`), nor again after a moment; another member, the provider's other models among them, is asked once, and with none left the agent gets Command Code's words (`TestCommandCodeZDRRefusalRestsNothing`, `TestZDRRefusalNeverRests`). Another vendor's refusal in the same words, without the code, still rests the model (`TestZDRRefusalIsCommandCodesCodeOnly`).
5. The last candidate's error reaches the agent, with one exception for a group (#1418, `patience.go`). When every member has failed in a way that passes (`busy`: 408, 500, 502, 503, 504 or 529 that isn't a proxy, a quota, response protection, or an upstream that answered a stream request with a whole reply, which fails the same however often it is asked (`didNotStream`; the same-one retries of `passing` leave it alone too, `TestStreamlessMemberIsAskedOnce`), or a 429 that is a short rate limit, not a plan or credit), and nothing of the reply has been sent, the members that failed so are asked again in turn after a pause (`replan`). The pause is `retryPause` doubled each round up to `replanLongest()` (30 times `retryPause`, 30s), and never shorter than the soonest Retry-After; a member whose Retry-After ends past the pause is asked after the others. This goes on while the group's `Patience` lasts. A member of a group in the group uses the innermost that sets `Patience` (`innerPatience`, #2410), as FirstToken (#2246) and Sink (#2255) do; unset falls back to the group asked for, and an inner `PatienceOff` skips replan for that member (`TestNestedGroupPatienceThroughOuter`, `TestNestedGroupPatienceInnerOff`, `TestNestedGroupPatienceFallsBackToOuter`). `replanRounds` (12) is only a backstop that never comes first: the pauses alone add up to `PatienceLongest` in 11 rounds (with 6, a group set to 240s gave up after 90s of pauses, real-jacket on #1418; `TestPatienceIsUsedUp`). The budget is the group's `Patience`, counted from the first time every member had failed, when the agent would have had the error (`waitFrom`; `Group.Waits`: 0 is `PatienceDefault`, 60s; -1, `PatienceOff`, turns it off; at most `PatienceLongest`, 240s). A pause that would end past the budget isn't taken, and the agent gets the error. While another member is busy too, the last one isn't first tried again alone (`passing`): the members take turns, and once the budget has begun, the same-one retries aren't taken either. While the budget lasts, the last candidate's failure is held (`newHoldWriter`), and a streaming agent is kept alive through the pause with SSE comments (`pauseAlive`). The try is traced with `Again` (the pause), `Replan` (how many are asked again) and `Patience` (the budget in seconds). The group editor's "Every member busy" row sets it. A plain provider keeps only the same-one retries `passing` gives the last one left. An agent that goes away in either pause (the same-one retry's or a group's replan) ends the request there with 499: nobody is asked again, and the try whose pause it was is marked `Gone` (#1894, heng91: a relay down while its proxy was off, the agent's window closed). The Routing page says the agent canceled the request before the retry and lists it as canceled, rather than ending its story on "tried again in 2s", a retry that never came (`TestAgentGoneInTheRetryPause`, `routing-agent-gone.test.cjs`). A try that answered is recorded (`served`, `answered`) for routing, affinity and usage. A try is added to the route's `Tries` only as it is asked, so a later try is the record that the gateway went on from this one, and the Routing page tells what came of a try from that, never from where it stood in the order (#2281, xishuchkui-code: an Antigravity 502 empty reply that went on to the next account was told as "an error another account wouldn't fix, so Claude Code gets it"). The one case the record can't show is the last try left out of its allowance after an earlier failure that wasn't one, where the agent is told the earlier error: that try is marked `Earlier` (`routing-went-on.test.cjs`).
## Constraints and failure behavior
@@ -876,7 +876,7 @@ the row of the entry that id reaches, with its label, under the bare id.
## Verification
```sh
-go test -tags nogui ./internal/gateway -run 'TestGroupAsksOneRestingMemberNotEach|TestLoneRestingMemberStillAsked|TestRetrievalGroupAsksOneRestingMember|TestProbeOne|TestLoop|TestLongReplyThatDoesNotLoop|TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestModelNotInPlanRestsTheModelNotTheKey|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestCodexCredits|TestWindowCapsHoldTheAccount|TestAccountCap|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestSearchFollowsTheSearchersProxy|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestProtection|TestSearchedConversation|TestUnsearchedConversationUnchanged|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestClaudeAccountBackOnceItsWindowRenews|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestCodexHeardWithinItsIdleTimeout|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused|TestAIStudioThinking|TestGeminiThoughtsThroughLocalProxy|TestGeminiRefusedLevel|TestGeminiThinkingOffAtItsLowest|TestThinkingOff|TestTranslatedEffortLoggedAsSent|TestTranslatedThinkingOffRefused|TestSub2APIThinkingOffRefused|TestRefusesThinkingOff|TestClaudeFiveFiveThinkingOffOnTheWire|TestSamplingAsTheModelTakesIt|TestHaikuFiveFiveSamplingOnTheWire|TestGeminiThoughtsMarkedInExtraContent|TestGeminiThinkingLevelRefused|TestGeminiMinimalRefusedTwiceAtOnce|TestPassthroughGeminiMinimalRefused|TestPassthroughEffortNoneRefused|TestPassthroughMinimalRefusedNoneTaken|TestOtherOffNamed|TestClaudeSignInNotToldItsSignInFailed|TestSignInWriterReleasesA403|TestSignInBetaStaysWithMagpie|TestStructuredOutput|TestResponsesTextKeepsVerbosityBesideFormat|TestRefusedFormatToldInWords|RPM|TestQueued|TestCustomGeminiUpstream|TestWholeNums|TestGrokCalls|TestOnlyGrokCalls|TestGrokModel|TestGrokPluginCalls'
+go test -tags nogui ./internal/gateway -run 'TestGroupAsksOneRestingMemberNotEach|TestLoneRestingMemberStillAsked|TestRetrievalGroupAsksOneRestingMember|TestProbeOne|TestLoop|TestLongReplyThatDoesNotLoop|TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestTryNow|TestRenewedToldToTheRoutingPage|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestModelNotInPlanRestsTheModelNotTheKey|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestCodexCredits|TestWindowCapsHoldTheAccount|TestAccountCap|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestSearchFollowsTheSearchersProxy|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestProtection|TestSearchedConversation|TestUnsearchedConversationUnchanged|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestClaudeAccountBackOnceItsWindowRenews|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestCodexHeardWithinItsIdleTimeout|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused|TestAIStudioThinking|TestGeminiThoughtsThroughLocalProxy|TestGeminiRefusedLevel|TestGeminiThinkingOffAtItsLowest|TestThinkingOff|TestTranslatedEffortLoggedAsSent|TestTranslatedThinkingOffRefused|TestSub2APIThinkingOffRefused|TestRefusesThinkingOff|TestClaudeFiveFiveThinkingOffOnTheWire|TestSamplingAsTheModelTakesIt|TestHaikuFiveFiveSamplingOnTheWire|TestGeminiThoughtsMarkedInExtraContent|TestGeminiThinkingLevelRefused|TestGeminiMinimalRefusedTwiceAtOnce|TestPassthroughGeminiMinimalRefused|TestPassthroughEffortNoneRefused|TestPassthroughMinimalRefusedNoneTaken|TestOtherOffNamed|TestClaudeSignInNotToldItsSignInFailed|TestSignInWriterReleasesA403|TestSignInBetaStaysWithMagpie|TestStructuredOutput|TestResponsesTextKeepsVerbosityBesideFormat|TestRefusedFormatToldInWords|RPM|TestQueued|TestCustomGeminiUpstream|TestWholeNums|TestGrokCalls|TestOnlyGrokCalls|TestGrokModel|TestGrokPluginCalls'
go test -tags nogui ./internal/gateway -run 'TestMemberTooSmallIsNotAsked|TestMemberTooSmallIsTheLastResort|TestClaudeAccountsSpentFallBackToCodex|TestConversationOutgrowingItsMemberMoves|TestRoomyLeavesWhatItCantJudge|TestPayloadTooLarge|TestOverflow|TestContextOverflow|TestRelayInputCap|TestEveryVendorsOverflow'
go test -tags nogui ./internal/gateway -run 'TestPlain403|TestOrderedKeepsAnAccountAt98|TestPromptRefusalRestsNobody'
go test -tags nogui ./internal/gateway -run 'TestVertex'
diff --git a/internal/gateway/fallback.go b/internal/gateway/fallback.go
index b02d35339..efb7ecf82 100644
--- a/internal/gateway/fallback.go
+++ b/internal/gateway/fallback.go
@@ -1011,7 +1011,40 @@ var restingUntil = struct {
sync.Mutex
m map[string]time.Time
note map[string]Rest // why each rests
-}{m: map[string]time.Time{}, note: map[string]Rest{}}
+ // lifted is when each rest was lifted before its time — by hand
+ // (Unrest), or its windows renewed — by the key it rested by: the
+ // kept routes still say it rested, and the Routing page reads this
+ // (TraceState.Lifted) to stop showing a rest lifted since it was
+ // found, from whichever page lifted it (zoecats on Discord: Try now
+ // in the provider's editor left Routing at 429 until a restart)
+ lifted map[string]time.Time
+}{m: map[string]time.Time{}, note: map[string]Rest{}, lifted: map[string]time.Time{}}
+
+// liftedLocked notes that the rest by key was lifted now, with
+// restingUntil held. A lift older than any rest can be is let go.
+func liftedLocked(key string, now time.Time) {
+ for k, at := range restingUntil.lifted {
+ if now.Sub(at) > longestMonthly {
+ delete(restingUntil.lifted, k)
+ }
+ }
+ restingUntil.lifted[key] = now
+}
+
+// liftedRests is when each rest lifted before its time was lifted, by
+// the key it rested by.
+func liftedRests() map[string]time.Time {
+ restingUntil.Lock()
+ defer restingUntil.Unlock()
+ if len(restingUntil.lifted) == 0 {
+ return nil
+ }
+ out := make(map[string]time.Time, len(restingUntil.lifted))
+ for k, at := range restingUntil.lifted {
+ out[k] = at
+ }
+ return out
+}
func (s *Server) resting(id string) bool {
_, ok := restOf(id)
diff --git a/internal/gateway/routing.go b/internal/gateway/routing.go
index ab9170efa..bd2120156 100644
--- a/internal/gateway/routing.go
+++ b/internal/gateway/routing.go
@@ -523,10 +523,12 @@ type Rest struct {
func renewed(agent, user string) {
restingUntil.Lock()
defer restingUntil.Unlock()
+ now := time.Now()
for k, r := range restingUntil.note {
if r.agent == agent && strings.EqualFold(r.user, user) && (r.Why == failQuota || r.By == "window") {
delete(restingUntil.m, k)
delete(restingUntil.note, k)
+ liftedLocked(k, now)
}
}
}
@@ -590,14 +592,19 @@ var (
func RestOf(key string) (Rest, bool) { return restOf(key) }
// Unrest lifts the rest of what rests by key — an account just verified
-// with its vendor, say — so the next request asks it again. False when it
-// wasn't resting.
+// with its vendor, say — so the next request asks it again: rate limited
+// with Sink on, it is no longer sent to the back of the order either.
+// The lift is told to the Routing page (TraceState.Lifted), wherever it
+// was asked for. False when it wasn't resting.
func (s *Server) Unrest(key string) bool {
restingUntil.Lock()
- defer restingUntil.Unlock()
_, ok := restingUntil.m[key]
delete(restingUntil.m, key)
delete(restingUntil.note, key)
+ liftedLocked(key, time.Now())
+ restingUntil.Unlock()
+ unsink(key)
+ s.trace.lifted()
return ok
}
@@ -606,15 +613,19 @@ func (s *Server) Unrest(key string) bool {
// asked again for all of them. It returns how many it lifted.
func (s *Server) UnrestAll(key string) int {
restingUntil.Lock()
- defer restingUntil.Unlock()
- n := 0
+ n, now := 0, time.Now()
for k := range restingUntil.m {
if k == key || strings.HasPrefix(k, key+"/") {
delete(restingUntil.m, k)
delete(restingUntil.note, k)
+ liftedLocked(k, now)
n++
}
}
+ liftedLocked(key, now)
+ restingUntil.Unlock()
+ unsink(key)
+ s.trace.lifted()
return n
}
diff --git a/internal/gateway/sink.go b/internal/gateway/sink.go
index ff9e5d262..5f1010b2b 100644
--- a/internal/gateway/sink.go
+++ b/internal/gateway/sink.go
@@ -13,7 +13,8 @@ package gateway
// that one's rest ends.
//
// What sank is remembered in memory, until magpie restarts — or until it
-// sinks again, which sends it to the back once more. Nothing else changes:
+// sinks again, which sends it to the back once more, or its rest is
+// lifted by hand (Unrest), which puts it back in its place. Nothing else changes:
// a resting one still waits behind every one ready (restLast), an account
// out of quota or credit rests as before and doesn't sink, and with Sink
// off the order is the routing's alone.
@@ -45,6 +46,15 @@ func sink(key string, now time.Time) {
sunk.Unlock()
}
+// unsink takes what rests by key off the back of the orders that sink,
+// its rest lifted by hand (Unrest): the next request asks it in its
+// place, as the lift says.
+func unsink(key string) {
+ sunk.Lock()
+ delete(sunk.m, key)
+ sunk.Unlock()
+}
+
// sunkAt is when a candidate sank, if it did.
func sunkAt(c candidate) (sinking, bool) {
sunk.Lock()
diff --git a/internal/gateway/trace.go b/internal/gateway/trace.go
index 5abc2d074..0b6389659 100644
--- a/internal/gateway/trace.go
+++ b/internal/gateway/trace.go
@@ -491,6 +491,17 @@ type trace struct {
// folders: each session's folder, as its requests said it last, for
// the requests that don't say it (a title, a topic check)
folders map[string]sessionFolder
+ // liftSeq is the seq a rest was last lifted by hand at (lifted): a
+ // page waiting hears of it, though no route changed
+ liftSeq int64
+}
+
+// lifted tells the pages waiting that a rest was lifted (Unrest).
+func (t *trace) lifted() {
+ t.mu.Lock()
+ t.changed()
+ t.liftSeq = t.seq
+ t.mu.Unlock()
}
type sessionFolder struct {
@@ -518,6 +529,10 @@ type TraceState struct {
// (OwnerTrace): the console names it from here, so the name follows the
// trace when another magpie takes the gateway over (#2606)
By *TraceBy `json:"by,omitempty"`
+ // Lifted is when each rest lifted before its time was lifted, by the
+ // key it rested by (Rest.Key): the routes kept from before still say
+ // it rested, and the page shows it resting no more
+ Lifted map[string]time.Time `json:"lifted,omitempty"`
}
// TraceBy names the magpie a trace was read from.
@@ -712,7 +727,7 @@ func (s *Server) gatewayTrace(w http.ResponseWriter, r *http.Request) {
wait = 25 * time.Second
}
st := s.Trace(r.Context(), after, wait)
- writeJSON(w, 200, map[string]any{"object": "trace", "now": time.Now(), "seq": st.Seq, "routes": st.Routes, "totals": st.Totals, "ids": st.IDs, "pid": os.Getpid(), "serves": Serves})
+ writeJSON(w, 200, map[string]any{"object": "trace", "now": time.Now(), "seq": st.Seq, "routes": st.Routes, "totals": st.Totals, "ids": st.IDs, "lifted": st.Lifted, "pid": os.Getpid(), "serves": Serves})
}
// ownerURL is the gateway's address on loopback, where the magpie serving
@@ -772,18 +787,19 @@ func OwnerTrace(ctx context.Context, after int64, wait time.Duration) (TraceStat
return TraceState{}, false
}
var out struct {
- Object string `json:"object"`
- Seq int64 `json:"seq"`
- Routes []Route `json:"routes"`
- Totals Totals `json:"totals"`
- IDs []int64 `json:"ids"`
- Pid int `json:"pid"`
- Serves string `json:"serves"`
+ Object string `json:"object"`
+ Seq int64 `json:"seq"`
+ Routes []Route `json:"routes"`
+ Totals Totals `json:"totals"`
+ IDs []int64 `json:"ids"`
+ Lifted map[string]time.Time `json:"lifted"`
+ Pid int `json:"pid"`
+ Serves string `json:"serves"`
}
if decodeJSON(b, &out) != nil || out.Object != "trace" || out.Routes == nil {
return TraceState{}, false
}
- return TraceState{Seq: out.Seq, Routes: out.Routes, Totals: out.Totals, IDs: out.IDs, By: &TraceBy{Pid: out.Pid, Serves: out.Serves}}, true
+ return TraceState{Seq: out.Seq, Routes: out.Routes, Totals: out.Totals, IDs: out.IDs, Lifted: out.Lifted, By: &TraceBy{Pid: out.Pid, Serves: out.Serves}}, true
}
// OwnerLanes reads the lanes of the magpie serving the gateway's address, for
@@ -840,8 +856,9 @@ func (s *Server) Trace(ctx context.Context, after int64, wait time.Duration) Tra
st.Routes = append(st.Routes, c)
}
}
- if len(st.Routes) > 0 || wait <= 0 {
+ if len(st.Routes) > 0 || wait <= 0 || t.liftSeq > after {
t.mu.Unlock()
+ st.Lifted = liftedRests()
return st
}
if t.wake == nil {
@@ -862,6 +879,7 @@ func (s *Server) Trace(ctx context.Context, after int64, wait time.Duration) Tra
gather.Stop()
}
case <-deadline.C:
+ st.Lifted = liftedRests()
return st
case <-ctx.Done():
return st
diff --git a/internal/gateway/try_now_test.go b/internal/gateway/try_now_test.go
new file mode 100644
index 000000000..0f8dba711
--- /dev/null
+++ b/internal/gateway/try_now_test.go
@@ -0,0 +1,143 @@
+package gateway
+
+import (
+ "context"
+ "net/http"
+ "slices"
+ "testing"
+ "time"
+
+ "github.com/yetone/magpie/internal/provider"
+)
+
+func forgetLifts(t *testing.T) {
+ t.Helper()
+ clear := func() {
+ restingUntil.Lock()
+ restingUntil.lifted = map[string]time.Time{}
+ restingUntil.Unlock()
+ }
+ clear()
+ t.Cleanup(clear)
+}
+
+// zoecats on Discord: an account rate limited (429) with Sink on, lifted
+// with Try now in the provider's editor, is asked again in its place, not
+// left at the back of the order until magpie restarts.
+func TestTryNowPutsASunkAccountBack(t *testing.T) {
+ forgetSinks(t)
+ forgetLifts(t)
+ old := allowances
+ defer func() { allowances = old }()
+ allowances = func(string) map[string]provider.Allowance {
+ out := map[string]provider.Allowance{}
+ for _, u := range []string{"[email protected]", "[email protected]"} {
+ out[u] = provider.Allowance{{Used: 10, Span: 30 * 24 * time.Hour, Resets: time.Now().Add(24 * time.Hour)}}
+ }
+ return out
+ }
+ s := New()
+ acct := func(user string) candidate {
+ return candidate{p: provider.Provider{ID: "workbuddy", Routing: provider.Ordered, Account: &provider.Account{Agent: "workbuddy", User: user}}, model: "m", rest: "workbuddy@" + user}
+ }
+ cs := []candidate{acct("[email protected]"), acct("[email protected]")}
+ p := provider.Provider{ID: "workbuddy", Routing: provider.Ordered, Sink: true}
+ order := func() []string {
+ out, _ := weigh(p, slices.Clone(cs), "m", "")
+ out, _ = restLast(out, planned{order: make([]Weighed, len(out))})
+ return whoOf(out)
+ }
+ rate := []byte(`{"error":{"message":"Too many requests","type":"rate_limit_error"}}`)
+ if r := s.restAfter(cs[0], 429, http.Header{}, rate); r.Why != failRate {
+ t.Fatalf("a: %s", r.Why)
+ }
+ if got := order(); !slices.Equal(got, []string{"[email protected]", "[email protected]"}) {
+ t.Fatalf("rate limited: order %v", got)
+ }
+ if !s.Unrest(cs[0].restKey()) {
+ t.Fatal("Unrest: a wasn't resting")
+ }
+ if got := order(); !slices.Equal(got, []string{"[email protected]", "[email protected]"}) {
+ t.Fatalf("after Try now: order %v, want a back in its place", got)
+ }
+
+ // the same through UnrestAll (a key's value replaced)
+ s.restAfter(cs[0], 429, http.Header{}, rate)
+ s.UnrestAll(cs[0].restKey())
+ if got := order(); !slices.Equal(got, []string{"[email protected]", "[email protected]"}) {
+ t.Fatalf("after UnrestAll: order %v", got)
+ }
+}
+
+// A rest lifted before its time is told to the Routing page, wherever it
+// was lifted: the trace says when, by the rest's key, and a page waiting
+// on the trace hears of it at once though no route changed. Before, the
+// page kept showing the routes' old 429 until magpie restarted.
+func TestTryNowToldToTheRoutingPage(t *testing.T) {
+ forgetSinks(t)
+ forgetLifts(t)
+ s := New()
+ key := "fakeco#8a197f6f60"
+ restingUntil.Lock()
+ restingUntil.m[key] = time.Now().Add(10 * time.Minute)
+ restingUntil.m[key+"/m1"] = time.Now().Add(10 * time.Minute)
+ restingUntil.Unlock()
+
+ st := s.Trace(context.Background(), 0, 0)
+ if len(st.Lifted) != 0 {
+ t.Fatalf("lifted before any lift: %v", st.Lifted)
+ }
+ seq := st.Seq
+
+ woke := make(chan TraceState, 1)
+ go func() { woke <- s.Trace(context.Background(), seq, time.Minute) }()
+ // the waiter is waiting once it made the wake channel
+ for {
+ s.trace.mu.Lock()
+ w := s.trace.wake != nil
+ s.trace.mu.Unlock()
+ if w {
+ break
+ }
+ time.Sleep(time.Millisecond)
+ }
+ before := time.Now()
+ s.Unrest(key)
+ select {
+ case st = <-woke:
+ case <-time.After(30 * time.Second):
+ t.Fatal("a page waiting on the trace didn't hear of the lift")
+ }
+ if at, ok := st.Lifted[key]; !ok || at.Before(before) {
+ t.Fatalf("trace lifted %v, want %s at or after %s", st.Lifted, key, before)
+ }
+ if st.Seq <= seq {
+ t.Fatalf("seq %d, not past %d", st.Seq, seq)
+ }
+ // read again from where it is, it doesn't wake on the same lift
+ if again := s.Trace(context.Background(), st.Seq, 0); again.Seq != st.Seq {
+ t.Fatalf("seq moved without a change: %d to %d", st.Seq, again.Seq)
+ }
+
+ // UnrestAll tells each model's rest too
+ s.UnrestAll(key)
+ st = s.Trace(context.Background(), 0, 0)
+ if _, ok := st.Lifted[key+"/m1"]; !ok {
+ t.Fatalf("UnrestAll: lifted %v, want %s/m1", st.Lifted, key)
+ }
+}
+
+// A quota rest lifted when the account's windows renewed is told too.
+func TestRenewedToldToTheRoutingPage(t *testing.T) {
+ forgetSinks(t)
+ forgetLifts(t)
+ key := "codex@[email protected]"
+ restingUntil.Lock()
+ restingUntil.m[key] = time.Now().Add(time.Hour)
+ restingUntil.note[key] = Rest{Why: failQuota, agent: "codex", user: "[email protected]", Key: key}
+ restingUntil.Unlock()
+ renewed("codex", "[email protected]")
+ if _, ok := liftedRests()[key]; !ok {
+ t.Fatalf("renewed: lifted %v, want %s", liftedRests(), key)
+ }
+}
diff --git a/internal/gui/assets/plugins/routing/client.js b/internal/gui/assets/plugins/routing/client.js
index 0cb26b5de..641360932 100644
--- a/internal/gui/assets/plugins/routing/client.js
+++ b/internal/gui/assets/plugins/routing/client.js
@@ -311,6 +311,19 @@ function routingPage(part) {
// now is the gateway's time — or, in a replay, the time it is replaying
const now = () => rp ? rp.real : Date.now() + skew;
const at = (s) => new Date(s).getTime();
+ // when each rest was lifted by hand, by the key it rests by (#1869), on
+ // the gateway's clock: here, or anywhere else the trace tells of
+ // (d.lifted: the provider's editor's Try now, a key's new value)
+ const liftedAt = new Map();
+ function noteLifted(m) {
+ for (const [k, v] of Object.entries(m || {})) if (at(v) > (liftedAt.get(k) || 0)) liftedAt.set(k, at(v));
+ }
+ // lifted: the rest by key, found at found, was lifted since — by the
+ // time shown, so a replay of a request before the lift shows its rest
+ const lifted = (key, found) => { const l = liftedAt.get(key) || 0; return !!key && l >= found && l <= now(); };
+ // sunkKey is what a row sank by, as its rest's key is (restKey): an
+ // account by its provider and user, a key by its own id
+ const sunkKey = (w) => w.kind === "account" && w.who ? w.provider + "@" + w.who.toLowerCase() : w.id;
const known0 = (s) => s && !s.startsWith("0001-");
function dur(ms) {
const s = Math.max(1, Math.round(ms / 1000));
@@ -1902,9 +1915,15 @@ function routingPage(part) {
// rest of one model only: the account's other models' rows don't
// rest with it (#1960)
const r = src().get(row.rid) || pinned || cur, answered = new Set(), rests = new Map(), gave = new Map(), went = new Map();
- const restAt = (id, m, rest) => { const rm = restModel(rest, m); rests.set(rm ? id + "\u0000" + rm : id, rest); };
+ // a rest lifted since it was found (#1869), here or in the
+ // provider's editor, is over: the row doesn't say it until the next
+ // request is refused again (zoecats on Discord)
+ const restAt = (id, m, rest, found) => {
+ if (lifted(rest.key, found)) return;
+ const rm = restModel(rest, m); rests.set(rm ? id + "\u0000" + rm : id, rest);
+ };
const modelOf = (tr) => tried(r, tr)?.model || tr.model || "";
- for (const w of r.order) if (w.rest) restAt(w.id, w.model, w.rest);
+ for (const w of r.order) if (w.rest) restAt(w.id, w.model, w.rest, at(r.time));
for (const tr of r.tries) {
if (tr.done && tryOk(tr)) answered.add(tryseat(r, tr));
// the error the agent got: the last try's, once the request is
@@ -1913,7 +1932,7 @@ function routingPage(part) {
// one the gateway went on from says what it answered, and that the
// request went on, not that the agent got it
else if (tr.done && !tr.rest && !tr.again && !tr.credits && tr !== r.tries[r.tries.length - 1]) went.set(tryseat(r, tr), tr);
- if (tr.rest) restAt(tr.id, modelOf(tr), tr.rest);
+ if (tr.rest) restAt(tr.id, modelOf(tr), tr.rest, at(tr.start) + (tr.ms || 0));
}
// it answered: whatever rest it began in is over
for (const tr of r.tries) if (tr.done && tryOk(tr)) { rests.delete(tr.id); rests.delete(tr.id + "\u0000" + modelOf(tr)); }
@@ -1928,7 +1947,7 @@ function routingPage(part) {
else if (gave.has(id)) s = t("{status} · {fail} · passed to {agent}", { status: gave.get(id).status, fail: failWord(gave.get(id).fail), agent: agentName(r.agent) });
else if (went.has(id)) s = t("{status} · {fail} · went on to the next", { status: went.get(id).status, fail: failWord(went.get(id).fail) });
else if (w.noURL) s = noURLWord(w);
- else if (w.sunk) s = t("rate limited at {time} · at the back", { time: clock(w.sunk) });
+ else if (w.sunk && !lifted(sunkKey(w), at(w.sunk))) s = t("rate limited at {time} · at the back", { time: clock(w.sunk) });
else if (w.known) {
const soon = usedRenews(w);
s = !w.routing && w.used >= 98 && w.credits ? quota(w, "{n} used · on credits", "{n} left · on credits")
@@ -3085,8 +3104,6 @@ function routingPage(part) {
+ (x.resetsAt ? " · " + resetInText(new Date(x.resetsAt)) : "")) : [];
}
- // when each rest was lifted by hand, by the key it rests by (#1869)
- const liftedAt = new Map();
// modelRest keeps the latest word of one of an account's models resting
// alone (#1960)
function modelRest(a, m, rest, when) {
@@ -3132,8 +3149,8 @@ function routingPage(part) {
// a rest lifted by hand since the request that found it (#1869): the
// kept requests still say it, until the next one asks it again
for (const a of by.values()) {
- if (a.rest?.key && (liftedAt.get(a.rest.key) || 0) >= a.restAt) a.rest = null;
- for (const [m, x] of a.mrests) if (x.rest.key && (liftedAt.get(x.rest.key) || 0) >= x.at) a.mrests.delete(m);
+ if (a.rest && lifted(a.rest.key, a.restAt)) a.rest = null;
+ for (const [m, x] of a.mrests) if (lifted(x.rest.key, x.at)) a.mrests.delete(m);
}
const list = [...by.values()].sort((x, y) => (x.w.name || "").localeCompare(y.w.name || "") || x.w.provider.localeCompare(y.w.provider) || rankOf(x.w) - rankOf(y.w) || x.pos - y.pos);
setText(actLabel, t("Accounts and keys"));
@@ -3241,7 +3258,7 @@ function routingPage(part) {
if (rest.why !== "restricted" && rest.why !== "verify") go.title = t("Ends the rest: the next request asks it again. Refused again, it rests again.");
go.onclick = async () => {
try {
- for (const x of lifts) { await api("gateway/unrest", { key: x.key }); liftedAt.set(x.key, Date.now()); }
+ for (const x of lifts) { await api("gateway/unrest", { key: x.key }); liftedAt.set(x.key, Date.now() + skew); }
render();
} catch (e) { go.textContent = e.message; }
};
@@ -3824,6 +3841,7 @@ function routingPage(part) {
const d = await res.json();
heard = true;
noteAccounts(d);
+ noteLifted(d.lifted);
skew = at(d.now) - Date.now();
mine = d.mine;
// another magpie took the gateway over (#2606): its routes and seq
diff --git a/internal/gui/tests/routing-lifted-elsewhere.test.cjs b/internal/gui/tests/routing-lifted-elsewhere.test.cjs
new file mode 100644
index 000000000..474b088bb
--- /dev/null
+++ b/internal/gui/tests/routing-lifted-elsewhere.test.cjs
@@ -0,0 +1,116 @@
+// Run with Node's test runner and Playwright on the module path; see README.md.
+// A rest lifted outside the Routing page (zoecats on Discord): a key rate
+// limited (429) and another sent to the back of the order, then Try now
+// pressed in the provider's editor. Before, Routing kept saying 429 and
+// "at the back" on its stage and under Accounts and keys until magpie
+// restarted, as the requests it keeps still say so. Now the trace tells the
+// page what was lifted and when (lifted), and the page shows neither
+// resting, with no request since. In every language, at a wide and a
+// narrow window, in Chromium and WebKit.
+const assert = require("node:assert/strict");
+const fs = require("node:fs/promises");
+const path = require("node:path");
+const { test } = require("node:test");
+const { chromium, webkit } = require("playwright");
+
+const assets = path.resolve(__dirname, "../assets");
+const now = new Date();
+const start = new Date(now.getTime() - 60e3).toISOString();
+const key = (k, extra = {}) => ({ id: "relay#" + k, provider: "relay", name: "Relay", kind: "key", who: "key " + k, model: "glm-5", ...extra });
+const rest = { why: "rate", status: 429, until: new Date(now.getTime() + 10 * 60e3).toISOString(), key: "relay#a" };
+const a = key("a", { rest }), b = key("b"), c = key("c", { sunk: start });
+const routes = [{
+ id: 1, seq: 1, time: start, agent: "codex", model: "relay/glm-5", provider: "relay",
+ order: [b, a, c],
+ tries: [
+ { id: a.id, model: a.model, start, done: true, status: 429, ms: 300, fail: "rate", rest, error: "Relay: Too many requests" },
+ { id: b.id, model: b.model, start, done: true, status: 200, ms: 900 },
+ ],
+ done: true, status: 200, ms: 1300,
+}];
+
+function serve(lang, lift) {
+ return async (route) => {
+ const url = new URL(route.request().url());
+ const json = (data) => route.fulfill({ json: data });
+ if (url.pathname === "/boot.js") return route.fulfill({ contentType: "text/javascript", body: `window.bootPrefs = {lang:"${lang}",theme:"light",web:true};` });
+ if (url.pathname === "/wails/runtime.js") return route.fulfill({ contentType: "text/javascript", body: "export const Window = {};" });
+ if (url.pathname === "/api/state") return json({ agents: [{ id: "codex", name: "Codex", path: "/test/config.toml", fields: [] }], profiles: [], settings: { lang, theme: "light" } });
+ if (url.pathname === "/api/plugins") return json({ plugins: [] });
+ if (url.pathname === "/api/gateway/trace") {
+ if (!url.searchParams.has("wait")) return json({ mine: true, now: now.toISOString(), seq: 1, totals: { requests: 1, rerouted: 1, errors: 0 }, ids: [1], routes });
+ // the page waits; Try now in the provider's editor lifts both, and
+ // the trace tells it with no route of its own
+ if (lift.told) return; // the next wait waits on
+ lift.told = true;
+ await lift.when;
+ const t = new Date().toISOString();
+ return json({ mine: true, now: t, seq: 2, totals: { requests: 1, rerouted: 1, errors: 0 }, ids: [1], routes: [], lifted: { "relay#a": t, "relay#c": t } });
+ }
+ if (url.pathname === "/api/gateway/history") return json({ cut: false, days: [], routes: [] });
+ if (url.pathname === "/api/groups") return json({ groups: [] });
+ if (url.pathname === "/api/providers") return json({ providers: [], presets: [], excluded: [], gateway: { running: true, window: true } });
+ if (url.pathname.startsWith("/api/")) return json({});
+ const file = path.join(assets, url.pathname === "/" ? "index.html" : url.pathname);
+ const contentType = { ".html": "text/html", ".js": "text/javascript", ".css": "text/css", ".svg": "image/svg+xml", ".png": "image/png" }[path.extname(file)];
+ await route.fulfill({ body: await fs.readFile(file).catch(() => ""), contentType });
+ };
+}
+
+const sideways = (page) => page.evaluate(() => document.documentElement.scrollWidth > window.innerWidth + 1);
+
+for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium", "webkit"])) {
+ for (const lang of ["en", "zh", "zh-TW", "ja", "de"]) {
+ for (const width of [1100, 420]) {
+ test(`${engine} ${lang} ${width}px: a rest lifted in the provider's editor stops showing on Routing`, async (t) => {
+ const browser = await (engine === "webkit" ? webkit.launch() : chromium.launch({ channel: "chromium" }));
+ const context = await browser.newContext({ viewport: { width, height: 900 }, reducedMotion: "reduce" });
+ const page = await context.newPage();
+ page.setDefaultTimeout(5000);
+ const errors = [];
+ page.on("pageerror", (e) => errors.push(e.message));
+ let go;
+ const lift = { told: false, when: new Promise((r) => { go = r; }) };
+ await page.route("**/*", serve(lang, lift));
+ t.after(async () => {
+ go();
+ if (process.env.ARTIFACT_DIR) {
+ await fs.mkdir(process.env.ARTIFACT_DIR, { recursive: true });
+ await page.screenshot({ path: path.join(process.env.ARTIFACT_DIR, `${engine}-${lang}-${width}-lifted-elsewhere.png`), fullPage: true });
+ }
+ await browser.close();
+ });
+ await page.goto("http://magpie.test/?view=routing");
+ await page.locator(".rt-act").first().waitFor();
+
+ // the three keys fold under their provider; the fold says one rests
+ const fold = page.locator("li.rt-fold").first();
+ await fold.waitFor();
+ const foldSaid = (await fold.locator("em").textContent()).trim();
+ await fold.click();
+ const stage = (who) => page.locator("li", { has: page.locator(".who", { hasText: new RegExp(`^${who}$`) }) }).filter({ has: page.locator("code.mdl") });
+ await stage("key a").first().waitFor({ state: "attached" });
+ assert.equal(await stage("key a").count(), 1);
+ assert.equal(await stage("key c").count(), 1);
+ // before: a rests, c is at the back
+ assert(await stage("key a").evaluate((e) => e.classList.contains("rest")), "key a doesn't rest before the lift");
+ // Accounts and keys folds the three too: one of them rests
+ const actSaid = (await page.locator(".rt-act .st").first().textContent()).trim();
+ const sunkSaid = (await stage("key c").locator("em").textContent()).trim();
+ assert(sunkSaid, "key c says nothing before the lift");
+
+ go();
+ await page.waitForFunction(() => {
+ const li = [...document.querySelectorAll("li")].find((l) => l.querySelector("code.mdl") && l.querySelector(".who")?.textContent === "key a");
+ return li && !li.classList.contains("rest");
+ });
+ assert.notEqual((await page.locator(".rt-act .st").first().textContent()).trim(), actSaid, "Accounts and keys still counts key a resting");
+ assert.notEqual((await stage("key c").locator("em").textContent()).trim(), sunkSaid, "key c still at the back after the lift");
+ await fold.click(); // folded again, it counts none resting
+ assert.notEqual((await fold.locator("em").textContent()).trim(), foldSaid, "the fold still counts key a resting");
+ assert(!(await sideways(page)), "the page scrolls sideways");
+ assert.deepEqual(errors, []);
+ });
+ }
+ }
+}