Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
gateway: a 429 that says it is back in a minute is a rate limit waited out, not a used-up plan (zoecats on Discord)
yetonedf64b62525f9
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex 5d404b453..6a5b61da7 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -163,7 +163,7 @@ recorded as a prediction failure (`noPrediction`).- A rule's hours and days (`Rule.Time`, `provider.TimeWindow`) are read on `ruleClock` in the zone they were set in (`TimeWindow.Zone`, an IANA name; gaubeebangeel on X: magpie on a cloud server in UTC, its user in Beijing, the hours 8 off). The GUI saves the browser's zone when the hours or days are changed in the editor and leaves a rule's zone alone otherwise; the CLI (`magpie group rule add … time=09:00-18:00@Asia/Shanghai` or `tz=`) and the TUI save the one typed, or this computer's (`provider.LocalZone`: `$TZ`, `time.Local`, the `/etc/localtime` link) when it has an IANA name. A window with no zone, every one saved before zones were kept, stays on magpie's own clock. A zone magpie doesn't know is refused on save (`TimeWindow.clean`), and one in a file edited by hand makes the window never hold (`zoneOf`): the rule is as if absent, never always on and never hours off. Zones come from Go's embedded tzdata, so Windows knows them too. `/api/groups` gives magpie's clock (`clock`: its zone and offset in seconds), and when it differs from the hours' zone the Routing page tells the zone beside them, on the group's rules and in the editor (`zoneNote` in [`plugins/routing/client.js`](../../internal/gui/assets/plugins/routing/client.js)): the zone's name, or the server's offset for hours with none. `TestRuleTimeWindowInItsZone`, `TestRuleTimeWindowInItsZoneAtTurnStart`, `TestRuleTimeWindowUnknownZone`, `TestParseRuleTimeZone`, `TestGroupRuleTimeZoneCmd`, `routing-time-zone.test.cjs`.3. Candidates come from `planGroup` for a group or `plan` for a provider. They are filtered by `sealedReaders` and `allowedCandidates`. An account past its share is held out of the candidates (`capHeld`): a usage cap's share, each window at its own where the account gives it one (`AccountWindowCaps`, willz on Discord: 50% of the five hours, 40% of the week) and at the account's cap otherwise, or 100% of a window for a Codex account set not to spend its credits (`provider.HoldCaps`, `WindowCaps.Of`, `CodexNoCredits`). A ChatGPT account's call on `codex-auto-review`, which OpenAI counts in no window, is held by no usage cap, only by credits it may not spend (`freeOnChatGPT`, #1593, `TestAccountCapLetsAutoReviewThrough`); a Free plan's call is capped as any other (`TestAccountCapHoldsAutoReviewOnAFreePlan`). The account is held while any window is past its own share, and `cappedError` and the trace name the fullest such window's share and cap. When every candidate is held and one held for its credits spends its resets by itself, `autoReset` spends a reset on it and it is tried (`planned.held`); otherwise a capped or barred set returns `cappedError` or `barredError`. Then `affine` moves the conversation's last answerer first, unless it is resting or nearly used up. An `X-Magpie-Account` header (`AccountHeader`) pins the request to one account: only that account is tried, and when it can't answer the caller is told why.4. The candidate loop calls `attempt` on each candidate in turn, through a `holdWriter`. The writer holds the reply until it is clearly passing. A candidate that fails before anything is sent rests (`failure`): out of credit for 30 minutes, out of quota until reset, rate limited until retry-after, otherwise a minute and longer on repeats: doubled each time, up to `longestRetry` (10 minutes) for a failure and `longestRateRest` (30 minutes) for a rate limit back as soon as its rest is over, or up to the provider's own `restCap` in minutes (`Provider.RestCapOr`, at most `MaxQuotaRest`; `magpie provider rest-cap <id> 4h`, #2008: a free key down for hours was asked every 10 minutes). A time the vendor names stands over it as before (`TestOwnRestCap`). The failure count is kept per key or account (`restKey`), and any of its models' answers clears it (`served`). Each model's own count (`restID`) is kept beside it and cleared only by that model's answer (`servedCandidate`): once it is past the account's, a sibling model answered since it last did, so the failure is the model's and it climbs its own ladder and rests alone, its siblings still served (#2241, weizsw: an Antigravity seat's gemini-3.5-flash-lite rested a minute 41 times of 43 while gemini-3.8-flash kept answering; `TestModelLadderClimbsPastSiblingAnswers`). Every model failing with no answer between still climbs the account's ladder and rests the account, and a candidate naming no model has the account's count alone. `rest-cap <id> off` (`provider.RestCapOff`, `restCap: -1`) is for a relay or account pool whose unknown errors come now and then and pass by the next request (Kismet Alan on Discord): a failure that says nothing of why (`failOther`: a 5xx, a dropped connection) then rests nobody and isn't counted, its trace rest `by: "off"` with no time left. A 401 or 403, a model not served, an account's window full, a rate limit (magpie's own 30-minute cap), quota, credit and a restriction still rest it as before (`TestRestCapOff`). A provider the user added can be set never to rest for a limit instead (`provider.LimitRestOff`, `limitRestOff` in providers.json; the editor's Advanced fold, "Set it aside when out of quota or rate limited", ticked by default): a refusal `restAfterMarked` reads as out of quota, out of credit or rate limited, and any 429, then rests nothing, isn't counted or sunk, and is `by: "off"`; `restDrawer` rests no image model of it. The next request is sent to it as usual. It is for a relay whose limits pass in a moment (zoecats on Discord: a local aggregator said "quota exhausted" while it switched upstreams, and magpie answered itself for 30 minutes; `TestLimitRestOffKeepsAsking`, `TestLimitRestOffRestsNothingForALimit`). A 401, a model not served and a plain failure still rest it. The shortest cap otherwise is a minute, the first rest; `rest-cap <id> 30s` is turned away and names `off`. A 400, 403, 404 or 422 that says the model isn't served (`unservedWords`) and says nothing of quota, credit or a rate limit rests only that key's or account's model (`modelRefused`, keyed by `restID`, its backoff counted apart): its other models are still served first in order (#1235, SenseNova's 403 "model is not available in the current token plan"). A Google account's Code Assist (the Antigravity and Gemini CLI built-ins) answering 404 `NOT_FOUND` ("Requested entity was not found.") rests its model the same way, as its endpoint is the same for every model (`codeAssistModelMissing`; Packing1 on Discord: Argon, listed for a Google AI Pro account, answered 404 and rested the whole account; `TestCodeAssistMissingModelRestsTheModel`). A refusal for want of quota, credit or rate, for an account its vendor restricted, or for a verify it asks of its account there, from another computer's magpie (a Remote magpie provider, LAN share) rests only the model it was about (`remoteModelRest`, keyed by `restID`, in `restAfterMarked` and `rateRest`): that magpie serves each of its models from its own providers and rests them itself, so one out of quota there says nothing of the others (bigsteven on Discord: a 429 for one model rested every model on the remote for the 3 hours it named, so a group never moved on to the remote's other models; `TestRemoteMagpieQuotaRestsTheModel`, `TestRemoteMagpieRestrictedRestsTheModel`). Another magpie added by hand, as an OpenAI-compatible provider of the user's own name rather than the preset, is read as one by its reply: a magpie names the provider that answered or held it (`X-Magpie-Provider`, `providerHeader`; `noteMember`, also on a `quotaHeld` or `verifyHeld` answer), as no vendor does, and `keepRetry` notes it as `answeredThereHeader`, since this magpie writes its own provider into the same header of its reply and the note is kept from the client (lagrangepoint1813 on Discord: "Remote Magpie: Claude Code: You've hit your session limit" answered every model of it, from `quotaHeld`, for the five hours it named; `TestAnotherMagpieAddedByHandRestsTheModel`, asked there and held there). Its key refused, the computer unreachable, or the key's own limit there (`keyLimited`'s 429, which carries `X-Magpie-Limit-Reset`, `remoteKeyLimitHeader`), still rests it whole: #2077's reporter had a Claude Code session limit on the other magpie refuse its Qoder models here (`TestRemoteMagpieRestsTheModel`, two gateways over HTTP; `TestRemoteMagpieKeyLimitRestsTheKey`). A 400, 404, 410 or 422 that says the model is retired (`retiredWords`: OpenCode Zen's 410 `ModelDeprecated` "Model exo-free has been deprecated.", whose own model list still names it; MOMO on Discord) rests only that model the same way, is retryable so a group's next member or another provider of it is asked, and is remembered (`provider.Retire`, `retired.json` beside `providers.json`): the model leaves that provider's entries in the catalog (`buildEntries`), so /v1/models, every agent's model list and the found groups go without it, until a request on it is answered again (`Unretire` in `servedCandidate`) or a week has passed. A refresh of the vendor's list doesn't bring it back early, since that list is where it still is. A `provider/model` request for it is still sent. A 400 or 422 that says this upstream's API can't take the request's shape (`shapeWords`: xAI's 422 "unknown item type" (#350), OpenAI's "Unknown parameter", and a limit of a model's own on tool schemas, OpenCode Go's 400 "JSON schema exceeds the maximum nesting depth of 10 levels" on muse-spark-1.3-contributor, #1870), or that it is served on another API than the one asked (`wrongEndpoint`, once the member has tried every API it has a URL for: #2670, a relay with a Chat URL alone refusing tools with reasoning and `none` too), rests nobody: the next candidate is asked once (`shapeRefused`, `failShape`), and the last one's error reaches the agent. A request missing what every API requires ("field required") isn't one, nor is a 200 that mentions it. Command Code's Provider API turning a request away with nothing but a trace id (`unsaidWords`: `{"message":"invalid request error trace_id: <hex>","type":"invalid_request_error"}`, a 400, or a 422 before) is handled the same way: it turns away a request whose history and asked-for output together are over a limit below the model's listed context (the reporter's A/B: one ~748K-token agent conversation answered 200 with `max_completion_tokens` 64000 and this 400 with 384000, the model's listed max output), which another member's model may take, and the key is fine (#2313, `TestGroupFailsOverCommandCodesBare400`). Its overflow, which names the model's context length before the trace id, and an error that names a `param`, aren't one. A relay's own cap on one request's input that says so (`tooLongRe`: B.ai's 400 "Input token exceed the limit", on a 380K request under the model's 1M window; #2445, worksyang) is taken as an overflow (`overflowed`), which no word list of `retryable` matched: a group's next member that may hold the request is asked (`withRoom`, `failOverflow`), nobody rests, another account of the same model isn't asked, and with no member left the agent is told the conversation is too long, so it compacts. A rate limit worded with input tokens (`paceWords`) isn't one, and a plain bad request is still asked of nobody else (`TestRelayInputCapGoesToTheNextMember`). A subscription account that fails any other way with a window full, by magpie's latest reading of its windows, rests until that window renews. A 401 or 403 is the exception: one that names a quota is out of quota already (`usedUpWords`), and one that doesn't is about the key or the request, so it rests on the backoff ladder whatever the reading says (plugins#243, zygs1083-dotcom: a Factory account read at 98% sat out its week over one plain 403; `TestPlain403DoesNotRestUntilTheWindow`). A plain 403 (`plainForbidden`: no quota, credit, rate, verification, restriction or model not served) from a second key or account of the same provider, model and reasoning in one request is the request refused, not each account: neither rests, the first one's rest and counted failure are taken back (`forbiddenAlike`, `unforbid`), and the provider's other mates are asked last (`matesLast`). Before, that one request walked every account of the provider into backoff (`TestPlain403OnTwoMatesRestsNeither`). One account's plain 403 alone still rests it. Both quota rests and window rests are held to `longestQuota` (8 days), so a reset time read wrong (a year off) doesn't bench the account for that long. The next candidate is then tried. A failed provider waits `fallbackCooldown` (1 minute) at the back. A WorkBuddy account the vendor restricted (#1849: 403 `{"code":11140,"msg":"request illegal","displayMsg":{"en":"Your account has been restricted due to unusual activity",…}}`) rests `restrictedRest` (2 hours) as `restricted` (`failRestricted`), not the minutes of an unknown failure: only WorkBuddy lifting the restriction, after the user's appeal, brings it back. Both shapes are known: the built-in's `wbExplain` and the plugin's `explained()` (0.1.17, the mover's `min`) end the error with `provider.WBRestrictedHint` (`restrictedSaid`), and the vendor's own body, its code 11140 or its words (`provider.WBRestricted`), counts only from a WorkBuddy candidate (`restrictedRefused`, `isWorkBuddy`), as another vendor's 11140 is its own. Bare "request illegal" isn't matched. WorkBuddy's 11128 channel refusal stays `promptRefused` and rests nobody. The Routing page names it "account restricted by the vendor", says the appeal is the way back, and gives the account's row a button that lifts the rest (`/api/gateway/unrest`) once the vendor has. Every other rest a key or account takes (quota, rate limit, retry-after, a failure's backoff) can be lifted by hand the same way (#1869): "Try now" on its row on the Routing page and beside its rest note in the provider's key list. It ends the rest, and with Sink on puts the key back from the back of the order; the backoff count stays, and a refusal after it rests the key again the usual way. Every lift before its time (Try now, `UnrestAll` for a replaced key, `renewed` for windows that renewed) is noted by its rest key (`restingUntil.lifted`) and handed to the Routing page in the trace (`TraceState.Lifted`, `lifted` in `/api/gateway/trace`), and a lift wakes a page waiting on the trace though no route changed (`trace.lifted`). The page stops showing a rest or a sink found before its lift, wherever the lift was asked for (zoecats on Discord: Try now in the provider's editor left Routing at 429 until a restart; `TestTryNowToldToTheRoutingPage`, `routing-lifted-elsewhere.test.cjs`). With another magpie serving the gateway, Try now lifts nothing (#3555). The vendor's safety filter refusing a reply rests nobody, whether it refuses the turn before any of the reply is said or once some of it has been said (`refused`, `refusedAfter`). So does one said in a stream a client is answered whole from (a Codex account is streamed upstream for a `stream:false` Responses, Chat or Messages request): each decoder gives the refusal's error event status 400 and its code (`filterRefused`), a whole reply it cut is answered the refusal rather than its words so far, and Anthropic's error shape, which has no code, carries it in `refusedHeader`; before, it was answered 502 `api_error`, asked again, and each account rested (#3274). `restAfterMarked` itself rests nobody, and counts no failure, for a 400 whose error object `policyRefusal` reads as a refusal, by its code, never by words a body may echo of the request. Its note on the try (and in the error the agent gets when nobody is left) is the vendor's reason, with Anthropic's `stop_details` category and explanation when it gave them (`refusedDetail`, #1768: "safety filter: refusal (category cyber)"); none is made up when it gave none. `stop_details` is carried through the IR (`Event.StopDetails`), so a Claude subscription run through the claude CLI and a translated Anthropic route keep it for an Anthropic client too. How many more candidates a refusal before any of the reply is asked of is Settings' `RefusalRetries` (`refusalAsksAgain`, counted per request in the candidate loop; a group's other members count too): unset, every one left (#248); 0, none (#1925, DD on Discord). When nobody is left, an agent on Anthropic's Messages API gets the held refusal itself, the vendor's 200 with `stop_reason` `refusal` and its `stop_details`, streamed or whole (`holdWriter.handBack`), since that is what Claude Code switches to its fallback model on; a 400 is only an API error to it. Codex and Chat clients still get `failTo`'s 400 (`refusedStatus`), which they don't ask again for. `TestRefusalRetries*` in [`refusal_retries_test.go`](../../internal/gateway/refusal_retries_test.go) cover keys, a group and the Claude subscription. The ChatGPT backend's 502 "response protection is unavailable" is about the request's content, not the account: the same history fails the same on each of its accounts and models. It rests nobody, isn't tried again after a moment, and isn't asked of the provider's other accounts or models; another provider left in a group is asked once (`protectionRefused`, `elsewhere`). One cause is known (#1270): a history that replays the backend's own `web_search_call` in a request that doesn't declare the `web_search` tool is refused so, whatever the account, model or turn (a plain turn, a thread's description, a compaction, Codex's own text compaction for a custom provider with `tools: []`). Every request to the ChatGPT backend with such a history therefore declares cached-only search (`{"type":"web_search","external_web_access":false}`, `provider.DeclareSearch`): an account's in `codexBody`, Codex's own sign-in relayed in `codexUpstreamOn`. It goes in the tools, or for Responses Lite (its header, `reasoning.context` `all_turns`, or an `additional_tools` item already there), which refuses a top-level `web_search`, in the `additional_tools` item. A request that offered no tools gets `tool_choice` `none` so the model can't search; one with a forced choice keeps it. A history without a search, or a request that declares search already, is relayed unchanged; Codex's own `web.run` (a function in the `web` namespace, which Codex offers in place of the hosted tool on a Responses Lite model or with its standalone web search on, and runs through `/alpha/search`) counts as declaring it, since the backend refuses a request carrying both `web.run` and the hosted `web_search` (Savin on Discord, Codex 0.163.0-alpha; `TestSearchedConversationUnderWebRun`). siwc, API-key requests to openai.com and the legacy `/responses/compact` are left as they were. Command Code turning a model away for its Zero Data Retention setting (its plugin's switch, 0.1.14, which sends `x-cmd-zdr: 1`; magpie-community/plugins#246) is a refusal of that request's setting, not of the model or the account: the Provider API's 422 with `"code":"cmd_zdr_no_providers"` and the 400 the plugin makes of Go's error line with `"type":"cmd_zdr_no_providers"` (`zdrCoded`, matched as the error's code or type, never by its words, whose "…a different model.","type":"invalid_request_error" read as a model unserved and rested the model on the account). `keepRetry` notes it beside the error (`zdrHeader`), since the error the agent gets may keep no code (Anthropic's shape, or the 400's numeric code). It rests nobody, anywhere (`restAfterMarked` returns `by: "off"`), so the model answers as soon as the switch is off; it isn't asked of the provider's other accounts at that model, which carry the same header (`notMates`), nor again after a moment; another member, the provider's other models among them, is asked once, and with none left the agent gets Command Code's words (`TestCommandCodeZDRRefusalRestsNothing`, `TestZDRRefusalNeverRests`). Another vendor's refusal in the same words, without the code, still rests the model (`TestZDRRefusalIsCommandCodesCodeOnly`).-5. The last candidate's error reaches the agent, with one exception for a group (#1418, `patience.go`), which a plain provider shares once its same-one retries are spent (below). When every member has failed in a way that passes (`busy`: 408, 500, 502, 503, 504 or 529 that isn't a proxy, a quota, response protection, or an upstream that answered a stream request with a whole reply, which fails the same however often it is asked (`didNotStream`; the same-one retries of `passing` leave it alone too, `TestStreamlessMemberIsAskedOnce`), or a 429 that is a short rate limit, not a plan or credit), and nothing of the reply has been sent, the members that failed so are asked again in turn after a pause (`replan`). The pause is `retryPause` doubled each round up to `replanLongest()` (30 times `retryPause`, 30s), and never shorter than the soonest Retry-After; a member whose Retry-After ends past the pause is asked after the others. This goes on while the group's `Patience` lasts. A member of a group in the group uses the innermost that sets `Patience` (`innerPatience`, #2410), as FirstToken (#2246) and Sink (#2255) do; unset falls back to the group asked for, and an inner `PatienceOff` skips replan for that member (`TestNestedGroupPatienceThroughOuter`, `TestNestedGroupPatienceInnerOff`, `TestNestedGroupPatienceFallsBackToOuter`). `replanRounds` (12) is only a backstop that never comes first: the pauses alone add up to `PatienceLongest` in 11 rounds (with 6, a group set to 240s gave up after 90s of pauses, real-jacket on #1418; `TestPatienceIsUsedUp`). The budget is the group's `Patience`, counted from the first time every member had failed, when the agent would have had the error (`waitFrom`; `Group.Waits`: 0 is `PatienceDefault`, 60s; -1, `PatienceOff`, turns it off; at most `PatienceLongest`, 240s). A pause that would end past the budget isn't taken, and the agent gets the error. While another member is busy too, the last one isn't first tried again alone (`passing`): the members take turns, and once the budget has begun, the same-one retries aren't taken either. While the budget lasts, the last candidate's failure is held (`newHoldWriter`), and a streaming agent is kept alive through the pause with SSE comments (`pauseAlive`). The try is traced with `Again` (the pause), `Replan` (how many are asked again) and `Patience` (the budget in seconds). The group editor's "Every member busy" row sets it. A plain provider (a `provider/model` request, its accounts and fallbacks the candidates) first takes the same-one retries `passing` gives the last one left: `lastRetries` for a busy vendor, `rateRetries` for a rate limit, each pause `retryPause<<n` or a Retry-After of at most `longestPause` (8s). Past those, or for a Retry-After longer than `longestPause`, it is replanned the same way, its accounts taking turns as members do, for the provider's own `Patience` (`Provider.Patience`, the same values as a group's, set with `magpie provider set <id> patience=2m|off|default`; the provider editor shows no row for it and its save keeps it, `TestProviderSaveKeepsPatience`), unset `singlePatience` (`PatienceDefault`, 60s), counted from when the agent would have had the error, so it works with no setting and the user's value wins either way (`TestSingleProviderOwnPatienceWins`; ChatGPT's reset of a large request, `codexReset`, is left out: Codex is told it at its 503 with what may help, as `codexBusy` leaves it). A provider's own Patience is for a request to it alone: a group it is in, set or unset, keeps the group's (zoecats on Discord: a local relay's short 429 went straight to Claude Code once the few retries were spent). So a 429 whose Retry-After is 20s is waited out, the stream kept alive past `keepHeldAfter`; one of an hour, a quota, credit or plan limit, a 401 or 403, a refusal, response protection's 502 and `didNotStream` are not `busy` or don't fit the budget, and reach the agent at once as the vendor said them. Its tries are traced with `Again` alone, no `Replan` or `Patience`, so the Routing page tells it as the last one left asked again rather than as a group's members (`TestSingleProviderWaitsOutAShortRateLimit`, `TestSingleProviderPassesALongLimitThrough`, `TestSingleProviderAgentGoneInTheWait` in [`single_busy_test.go`](../../internal/gateway/single_busy_test.go)). Codex's own one-account relay (`codexUpstreamOn`, below) doesn't join it: its request is relayed unchanged outside the candidate loop, and the backend's 429 is always the plan's usage limit (`codexBusy` never retries one), so the short rate limit this waits out doesn't arise there; its overloads keep their `lastRetries` and then the subscription's fallbacks. An agent that goes away in either pause (the same-one retry's or a group's replan) ends the request there with 499: nobody is asked again, and the try whose pause it was is marked `Gone` (#1894, heng91: a relay down while its proxy was off, the agent's window closed). The Routing page says the agent canceled the request before the retry and lists it as canceled, rather than ending its story on "tried again in 2s", a retry that never came (`TestAgentGoneInTheRetryPause`, `routing-agent-gone.test.cjs`). A try that answered is recorded (`served`, `answered`) for routing, affinity and usage. A try is added to the route's `Tries` only as it is asked, so a later try is the record that the gateway went on from this one, and the Routing page tells what came of a try from that, never from where it stood in the order (#2281, xishuchkui-code: an Antigravity 502 empty reply that went on to the next account was told as "an error another account wouldn't fix, so Claude Code gets it"). The one case the record can't show is the last try left out of its allowance after an earlier failure that wasn't one, where the agent is told the earlier error: that try is marked `Earlier` (`routing-went-on.test.cjs`).+5. The last candidate's error reaches the agent, with one exception for a group (#1418, `patience.go`), which a plain provider shares once its same-one retries are spent (below). When every member has failed in a way that passes (`busy`: 408, 500, 502, 503, 504 or 529 that isn't a proxy, a quota, response protection, or an upstream that answered a stream request with a whole reply, which fails the same however often it is asked (`didNotStream`; the same-one retries of `passing` leave it alone too, `TestStreamlessMemberIsAskedOnce`), or a 429 that is a short rate limit, not a plan or credit), and nothing of the reply has been sent, the members that failed so are asked again in turn after a pause (`replan`). The pause is `retryPause` doubled each round up to `replanLongest()` (30 times `retryPause`, 30s), and never shorter than the soonest Retry-After; a member whose Retry-After ends past the pause is asked after the others. This goes on while the group's `Patience` lasts. A member of a group in the group uses the innermost that sets `Patience` (`innerPatience`, #2410), as FirstToken (#2246) and Sink (#2255) do; unset falls back to the group asked for, and an inner `PatienceOff` skips replan for that member (`TestNestedGroupPatienceThroughOuter`, `TestNestedGroupPatienceInnerOff`, `TestNestedGroupPatienceFallsBackToOuter`). `replanRounds` (12) is only a backstop that never comes first: the pauses alone add up to `PatienceLongest` in 11 rounds (with 6, a group set to 240s gave up after 90s of pauses, real-jacket on #1418; `TestPatienceIsUsedUp`). The budget is the group's `Patience`, counted from the first time every member had failed, when the agent would have had the error (`waitFrom`; `Group.Waits`: 0 is `PatienceDefault`, 60s; -1, `PatienceOff`, turns it off; at most `PatienceLongest`, 240s). A pause that would end past the budget isn't taken, and the agent gets the error; a member the vendor said is back within a second past its budget (read in whole seconds) is asked as the budget ends (`TestReporterResetFitsTheDefaultPatience`). While another member is busy too, the last one isn't first tried again alone (`passing`): the members take turns, and once the budget has begun, the same-one retries aren't taken either. While the budget lasts, the last candidate's failure is held (`newHoldWriter`), and a streaming agent is kept alive through the pause with SSE comments (`pauseAlive`). The try is traced with `Again` (the pause), `Replan` (how many are asked again) and `Patience` (the budget in seconds). The group editor's "Every member busy" row sets it. A plain provider (a `provider/model` request, its accounts and fallbacks the candidates) first takes the same-one retries `passing` gives the last one left: `lastRetries` for a busy vendor, `rateRetries` for a rate limit, each pause `retryPause<<n` or a Retry-After of at most `longestPause` (8s). Past those, or for a Retry-After longer than `longestPause`, it is replanned the same way, its accounts taking turns as members do, for the provider's own `Patience` (`Provider.Patience`, the same values as a group's, set with `magpie provider set <id> patience=2m|off|default`; the provider editor shows no row for it and its save keeps it, `TestProviderSaveKeepsPatience`), unset `singlePatience` (`PatienceDefault`, 60s), counted from when the agent would have had the error, so it works with no setting and the user's value wins either way (`TestSingleProviderOwnPatienceWins`; ChatGPT's reset of a large request, `codexReset`, is left out: Codex is told it at its 503 with what may help, as `codexBusy` leaves it). A provider's own Patience is for a request to it alone: a group it is in, set or unset, keeps the group's (zoecats on Discord: a local relay's short 429 went straight to Claude Code once the few retries were spent). So a 429 whose Retry-After is 20s is waited out, the stream kept alive past `keepHeldAfter`; one of an hour, a quota, credit or plan limit, a 401 or 403, a refusal, response protection's 502 and `didNotStream` are not `busy` or don't fit the budget, and reach the agent at once as the vendor said them. Its tries are traced with `Again` alone, no `Replan` or `Patience`, so the Routing page tells it as the last one left asked again rather than as a group's members (`TestSingleProviderWaitsOutAShortRateLimit`, `TestSingleProviderPassesALongLimitThrough`, `TestSingleProviderAgentGoneInTheWait` in [`single_busy_test.go`](../../internal/gateway/single_busy_test.go)). Codex's own one-account relay (`codexUpstreamOn`, below) doesn't join it: its request is relayed unchanged outside the candidate loop, and the backend's 429 is always the plan's usage limit (`codexBusy` never retries one), so the short rate limit this waits out doesn't arise there; its overloads keep their `lastRetries` and then the subscription's fallbacks. An agent that goes away in either pause (the same-one retry's or a group's replan) ends the request there with 499: nobody is asked again, and the try whose pause it was is marked `Gone` (#1894, heng91: a relay down while its proxy was off, the agent's window closed). The Routing page says the agent canceled the request before the retry and lists it as canceled, rather than ending its story on "tried again in 2s", a retry that never came (`TestAgentGoneInTheRetryPause`, `routing-agent-gone.test.cjs`). A try that answered is recorded (`served`, `answered`) for routing, affinity and usage. A try is added to the route's `Tries` only as it is asked, so a later try is the record that the gateway went on from this one, and the Routing page tells what came of a try from that, never from where it stood in the order (#2281, xishuchkui-code: an Antigravity 502 empty reply that went on to the next account was told as "an error another account wouldn't fix, so Claude Code gets it"). The one case the record can't show is the last try left out of its allowance after an earlier failure that wasn't one, where the agent is told the earlier error: that try is marked `Earlier` (`routing-went-on.test.cjs`).## Constraints and failure behavior@@ -200,6 +200,7 @@ recorded as a prediction failure (`noPrediction`).- A subscription account whose sign-in its vendor revoked (`provider.SignInRevoked`: 401 `token_revoked` and the like, #1793) fails as `failAuth`: the next member is asked at once, nothing rests, and the account is marked (`noteSignIn` in [`signin_revoked.go`](../../internal/gateway/signin_revoked.go)). Its later requests aren't sent while it has that credential: `forwardOnce` answers them with a 401 of magpie's own (`revokedReply`), so they too pass on. Compaction on an account of magpie's (`compactOn`) picks a marked one last. A key's provider answering the same words rests as before.- A ChatGPT 401 that says the token sent has expired and asks for a refresh (`provider.SignInExpired`, #2973) while its `exp` is ahead renews the account's sign-in once (`provider.NoteExpiredSignIn` from `noteSignIn`), keyed on the token sent, so the next request goes with a fresh one; nothing is marked, and the request fails over as any 401 does ([Providers and accounts](providers-accounts.md), A token called expired).- A sub2api key out of its own limit answers 429 `api key 7天限额已用完` (or `日限额`, `5小时限额`) as `rate_limit_exceeded`. It is out of quota (`plannedWords`), not rate limited: it rests till the window it filled renews when its windows as last read say so, else `quotaRest`, and its windows are read again (`staleKeyAllowance`). A reading that finds that window full no more (the limit raised in the relay's panel, or the key's usage reset) lifts the rest (`renewed`, told through `provider.OnRenewed`), as a Codex reset spent does an account's; the key's other rests stay. While it rests out of its windows, they are read again each time it is planned (`restingOf`, once a minute at most), so this holds where nothing weighs it by them: an ordered group. The relay out of accounts for everyone (503 `No available accounts`, or a later version's 429 `All available accounts are currently rate-limited`) says nothing of the key's windows: it backs off as any failure or rate limit does, never to the window. A key's windows count only its own spend, so other users draining the relay's accounts show up as those errors, not in the windows.+- A 429 that says in words it is back within `soonest` (`PatienceLongest`, 240s) is a rate limit (`failRate`), whatever plan words it has besides, unless it speaks of money (`creditWords`): `backSoon` reads `resetsAfter` (now with seconds and Chinese minutes, `soonWords`: "try again in 30 seconds", "请 1 分钟后重试") and `tryAgainIn`. A LAN aggregator's "Usage has exceeded the frequency limit. We will switch you to another model. Your usage limit for the original model will reset in 1 minutes." read as a plan used up ("usage limit") and reached the agent at once; now it is `busy`, rests the minute it says (`rateRest` reads `resetsAt`), and a lone provider waits it out within its patience (zoecats on Discord; `TestShortResetIsARateLimit`, `TestShortResetRestsWhatItSays`, `TestSingleProviderWaitsOutAFrequencyLimit`). One that says hours is still out of quota and not waited (`TestSingleProviderPassesALongLimitThrough`). "Frequency limit" is a rate word, as 频率 is.- A 5xx is a server error, not quota, unless the vendor's own quota code says so (`quotaCodes`: `usage_limit_reached`, `RESOURCE_EXHAUSTED`, Claude Code's `usage limit reached|<unix>`). A relay's 500 "当前模型 gpt-5.5 负载已经达到上限,请稍后重试" names its load, not the key's allowance: it was read as out of quota by `usedUpWords` (上限), which rested the key till a reset and kept a group's Patience from waiting it out (#1418). The quota words (`usedUpWords`) still apply to a 4xx.- A 429 that names in words when the account is back (WorkBuddy's "将在 2026-10-11 20:16:02 UTC+8重置" and "your usage will reset at 2026-10-11 12:37:53 UTC+8, …", #1669; Volcengine Ark's "You have exceeded the monthly usage quota. It will reset at 2026-10-25 23:59:59 +0800 CST.", #3251, #3276) is out of quota in either language (`resetsWhen`), and rests until that time (`resetsWritten`, read in the offset written). Only the reset phrasing right beside a whole date and time with its UTC/GMT offset, or its offset as Go prints one (four digits, then the zone's name, which isn't read), followed by 重置 or punctuation, is read, so a refusal cut short isn't read as UTC and another vendor's date in passing is no reset; a time past, not a real date, or further off than `longestSaid` isn't read: `longestWritten` (7 days), or `longestMonthly` (35 days) when the refusal says its allowance is a month's (`monthlyWords`: Ark's, Zhipu's 1310 "Weekly/Monthly Limit Exhausted"). Such a reset is the one rest out of quota held past `longestQuota`, to `longestMonthly`; a window read full or the user's own rest is still held to 8 days (`TestMonthlyWrittenResetRestsTheMonth`). Ark's windows as its AccessKey reads them (`volcPlanQuotas`) still reach only its usage card, not routing (#3251). Zhipu's 1308 and 1310 (Z.ai and bigmodel.cn, GLM Coding Plan: "Your limit will reset at 2026-10-15 20:12:13", "您的限额将在 … 重置") write the time with no zone; it is Beijing's, and is read as UTC+8 in those words only (`zhipuResets`, `resetsZhipu`), never when a digit, colon or zone of its own follows (Player on Discord: read by nothing, the key rested `quotaRest` and the refusal said "not asked again before 17:12" while Z.ai said 20:12; `TestZhipuRestsUntilItsReset`). One that points to the account's other models (`otherModels`: "切换其他模型", "switch to the other models") rests that model alone (`restID`). Out of credit or quota on a model that costs credits, where the subscription serves other models free (`catalog.Model.Free`: WorkBuddy's x0.00, Qoder's is_free, Cline's free models; `Provider.PaidBesideFree`), rests that model alone too: the free ones draw on no credits (Creat.wu on Discord, `TestOutOfCreditLeavesFreeModels`). A Credits window leaves the free models out the same way, so an account at 100% isn't ranked spent for them: the WorkBuddy plugin's lists them as `notModels` (0.1.23), and the built-in's scopes itself by the provider's last fetched list (`wbSpareFree`; with no list read, every model counts) (plugins#138, hheei: an account at 560 / 560 was still served deepseek-v4.1-flash; `TestWorkBuddyCreditsSpareFreeModels`). A free model the vendor refuses all the same (WorkBuddy refuses even 0-rate models to an account with no package) rests the account on its own refusal. Cline's daily limit of one free model, 429 "Daily free limit reached on model deepseek/deepseek-v4-flash. Try again in 23h 59m" (the words Cline's own clients read it by, `isClineFreeModelLimitMessage`), rests that model alone (`clineFreeLimit`, `clineModelLimited`, keyed by `restID`) until the time it names (`clineBackIn`, `tryAgainIn`, held to `longestWritten`; `resets`), from a provider whose requests reach Cline's API (`ClinePinnable`: the ClinePass preset, one at cline.bot, the Cline plugin's). The key's other models, free or the plan's, are still served, and the model out of its day is answered with its refusal without asking Cline again (#1398, wuasakura: one free model's 429 sent the whole Cline account to the back; `TestClineFreeLimitRestsTheModel`, `TestClineFreeLimitLeavesOtherModels`). Another vendor's same words, and a Cline 429 that doesn't name the model's limit, rest the key as before. WorkBuddy's 429 14018 "Credits exhausted" is out of credit (`brokeWords`), not a rate limit, on the built-in's raw body and the plugin's `insufficient_quota` alike (`TestCreditsExhaustedIsOutOfCredit`). On the rate-limit branch, a reset time the refusal names (`resetsAt`, or `resetsHeader` as `keepRetry` noted it) is believed up to `longestQuota`, as out of quota believes it, and the Routing page says it rests until then (`resets`); a `Retry-After` or a rate-limit reset header is still held to `longestWait` (an hour) (`rateRest`). The same rest runs for a plugin's requests: the WorkBuddy plugin passes these 429s on as they come (`TestPluginWrittenResetRests`). A JSON error body that escapes what isn't ASCII as `\uXXXX` (Python's json.dumps does by default) has its strings decoded before any word list (`creditWords`, `rateWords`, `plannedWords`, `usedUpWords`, `resetsWhen`, …) is matched (`wordsOf`), so an escaped 余额不足 or 用量 reads as an unescaped one does.- Antigravity's 429 for a model's quota used up writes how long until it is back as Go prints a duration: "You have exhausted your capacity on this model. Your quota will reset after 3h19m30s." (`quotaResetAfter`, read by `resetsAfter` beside Factory's "resets in …", held to `longestWritten`). The account (or, as `pooled` says, that model on it) rests that long (`resets`), and the agent's Retry-After says it when no account is left. Unread, it rested `quotaRest`, so the account first in order was asked again, and refused, every 15 minutes while the next ticked account had its quota (Dani Ariyanto on Discord, 10-24; `TestAntigravityQuotaRestsUntilItsReset`). Its other 429, "Resource has been exhausted (e.g. check quota).", names no time and rests `quotaRest` as before.diff --git a/internal/gateway/frequency_limit_test.go b/internal/gateway/frequency_limit_test.gonew file mode 100644index 000000000..aa8f41ad8--- /dev/null+++ b/internal/gateway/frequency_limit_test.go@@ -0,0 +1,131 @@+package gateway++import (+ "bufio"+ "net/http"+ "net/http/httptest"+ "strings"+ "testing"+ "time"++ "github.com/yetone/magpie/internal/provider"+)++// zoecats on Discord (10-27): their LAN aggregator's 429, word for word.+const frequencyLimited = "Usage has exceeded the frequency limit. We will switch you to another model. Your usage limit for the original model will reset in 1 minutes."++// A 429 that says it is back within a wait a provider's patience can+// cover is a rate limit, waited for, whatever plan words it has besides;+// one that says it is back in hours, or that the money is spent, is not.+func TestShortResetIsARateLimit(t *testing.T) {+ for body, want := range map[string]string{+ `{"error":{"message":"` + frequencyLimited + `","type":"rate_limit_error"}}`: failRate,+ frequencyLimited: failRate,+ `{"error":{"message":"Usage limit exceeded, try again in 30 seconds"}}`: failRate,+ `{"error":{"message":"Daily usage limit hit. Retry after 20s."}}`: failRate,+ `{"error":{"message":"用量超出频率限制,请 1 分钟后重试"}}`: failRate,+ `{"error":{"message":"额度已用完,30秒后再试"}}`: failRate,+ `{"error":{"message":"Usage has exceeded the frequency limit."}}`: failRate,+ // back in hours: the plan's, not waited+ `{"error":{"message":"Your usage limit for the original model will reset in 5 hours."}}`: failQuota,+ `{"error":{"message":"Your usage limit will reset in 0h 30min"}}`: failQuota,+ `{"error":{"message":"Daily free limit reached on model x. Try again in 23h 59m"}}`: failQuota,+ `{"error":{"message":"You've hit your usage limit."}}`: failQuota,+ // money spent, however soon it says to come back+ `{"error":{"message":"Insufficient balance. Please try again in 30 seconds"}}`: failCredit,+ `{"error":{"message":"余额不足,请充值后 1 分钟后重试"}}`: failCredit,+ `{"error":{"message":"You exceeded your current quota, please check your plan and billing details"}}`: failQuota,+ } {+ if got := failure(429, []byte(body)); got != want {+ t.Errorf("%s: %s, want %s", body, got, want)+ }+ if got := busy(429, []byte(body)); got != (want == failRate) {+ t.Errorf("%s: busy %v, want %v", body, got, want == failRate)+ }+ }+ // a 403 saying the same isn't a rate limit+ if got := failure(403, []byte(frequencyLimited)); got == failRate {+ t.Errorf("403: %s", got)+ }+}++// The reporter's 429 rests its provider about the minute it says, not+// the quarter hour of a plan used up.+func TestShortResetRestsWhatItSays(t *testing.T) {+ old := allowances+ defer func() { allowances = old }()+ allowances = func(string) map[string]provider.Allowance { return nil }+ s := &Server{}+ c := candidate{p: provider.Provider{ID: "lan"}, model: "m", rest: "lan#k"}+ clearRest(c.restKey())+ t.Cleanup(func() { clearRest(c.restKey()) })+ r := s.restAfter(c, 429, http.Header{}, []byte(`{"error":{"message":"`+frequencyLimited+`"}}`))+ if d := time.Until(r.Until); r.Why != failRate || d > time.Minute || d < 50*time.Second {+ t.Fatalf("rests %v by %s (%s), want about a minute as a rate limit", d, r.By, r.Why)+ }+}++// The reporter's 429, with the Retry-After magpie gives it from its words+// (keepRetry), is waited out within the minute a lone provider waits by+// default (singlePatience), and asked again when the vendor said: the+// vendor's whole minute is not a hair past the budget. A time a second and+// more past it is still given up on.+func TestReporterResetFitsTheDefaultPatience(t *testing.T) {+ h := http.Header{}+ keepRetry(h, http.Header{}, []byte(`{"error":{"message":"`+frequencyLimited+`"}}`))+ if h.Get("Retry-After") != "60" {+ t.Fatalf("Retry-After %q, want the minute it says", h.Get("Retry-After"))+ }+ c := candidate{p: provider.Provider{ID: "lan"}, model: "m"}+ waitFrom := time.Now()+ list := noteBusy(nil, c, h)+ next, wait, ok := replan(list, 0, waitFrom, singlePatience)+ if !ok || len(next) != 1 || wait < 59*time.Second || wait > time.Minute {+ t.Fatalf("replan %v %v %v: want the one asked again after its minute", next, wait, ok)+ }+ for _, past := range []time.Duration{1500 * time.Millisecond, time.Hour} {+ list := []busyTry{{c: c, ready: waitFrom.Add(singlePatience + past)}}+ if _, _, ok := replan(list, 0, waitFrom, singlePatience); ok {+ t.Errorf("back %v past the budget: asked again", past)+ }+ }+}++// A request to one provider whose vendor answers the reporter's 429 is+// asked again when it says and answered, not handed the 429. Its minute+// is two seconds here, the rest of its words as they were.+func TestSingleProviderWaitsOutAFrequencyLimit(t *testing.T) {+ said := strings.Replace(frequencyLimited, "reset in 1 minutes", "reset in 2 seconds", 1)+ u := &busyUpstream{replies: []busyReply{+ {code: 429, body: `{"type":"error","error":{"type":"rate_limit_error","message":"` + said + `"}}`},+ {code: 200, body: anthropicAnswer},+ }}+ s := busyProvider(t, u)+ scaled(t, 5*time.Second)+ gw := httptest.NewServer(s.Handler())+ t.Cleanup(gw.Close)+ start := time.Now()+ res, err := http.Post(gw.URL+"/v1/messages", "application/json", strings.NewReader(singleAsk(true)))+ if err != nil {+ t.Fatal(err)+ }+ defer res.Body.Close()+ var body strings.Builder+ rd := bufio.NewReader(res.Body)+ for {+ line, err := rd.ReadString('\n')+ body.WriteString(line)+ if err != nil {+ break+ }+ }+ if res.StatusCode != 200 || !strings.Contains(body.String(), "message_stop") || u.asked() != 2 {+ t.Fatalf("agent got %d %q, upstream asked %d times; want the answer on the second ask; %+v", res.StatusCode, body.String(), u.asked(), lastRoute(s).Tries)+ }+ if took := time.Since(start); took < 2*time.Second {+ t.Errorf("asked again after %v, before the 2s it said", took)+ }+ if r := lastRoute(s); len(r.Tries) != 2 || r.Tries[0].Fail != failRate || r.Tries[0].Again < 1900 {+ t.Errorf("tries %+v: want the first asked again as a rate limit after its 2s", r.Tries)+ }+}diff --git a/internal/gateway/patience.go b/internal/gateway/patience.goindex 94a1e9e61..87e965bb4 100644--- a/internal/gateway/patience.go+++ b/internal/gateway/patience.go@@ -122,6 +122,20 @@ func replan(list []busyTry, round int, waitFrom time.Time, fallback time.Duratio}now := time.Now()wait := min(retryPause<<(round+1), replanLongest())+ // a vendor's time is read in whole seconds: one back within a second+ // past a member's budget is asked as the budget ends, not given up on+ // (zoecats on Discord: "will reset in 1 minutes" against the minute a+ // lone provider waits by default, which the moment between the+ // failure and its note put a hair past)+ list = slices.Clone(list)+ for i, b := range list {+ if budget := waitsOf(b.c.patience, fallback); budget > 0 {+ end := waitFrom.Add(budget)+ if b.ready.After(end) && !b.ready.After(end.Add(time.Second)) {+ list[i].ready = end+ }+ }+ }soonest := list[0].readyfor _, b := range list[1:] {soonest = minTime(soonest, b.ready)@@ -134,7 +148,7 @@ func replan(list []busyTry, round int, waitFrom time.Time, fallback time.Duratioif budget <= 0 {continue}- if !at.Before(waitFrom.Add(budget)) {+ if at.After(waitFrom.Add(budget)) {continue // this member's budget is spent}alive = append(alive, b)diff --git a/internal/gateway/routing.go b/internal/gateway/routing.goindex 94d4acc8e..7e3e05fd7 100644--- a/internal/gateway/routing.go+++ b/internal/gateway/routing.go@@ -215,7 +215,8 @@ var (// exceeded", "reached"): as magpie words an upstream error, with// "rate_limit_error" for its type, every such 429 was, and rested a// quarter of an hour rather than a minute (#153).- rateWords = regexp.MustCompile(`(?i)rate.?limit|too many requests|per.?(second|sec|minute|min)\b|\b[rt]pm\b|频率|太频繁`)+ // "Frequency limit" is the English of 频率 (zoecats on Discord).+ rateWords = regexp.MustCompile(`(?i)rate.?limit|frequency.?limit|too many requests|per.?(second|sec|minute|min)\b|\b[rt]pm\b|频率|太频繁`)// plannedWords: a 429 that says the plan's own allowance is used, rate// words or not — a day's free requests, say, or a sub2api key's own// 5-hour, day or 7-day limit ("api key 7天限额已用完", sent as@@ -403,6 +404,15 @@ func failure(status int, body []byte) string {// English one as a rate limit, and each rested its own fallback// (#1669)return failQuota+ case status == 429 && !creditWords.Match(body) && backSoon(body):+ // back within a wait a provider's patience can cover, as it says+ // itself, whatever else it says: a LAN aggregator's "Usage has+ // exceeded the frequency limit. … Your usage limit for the+ // original model will reset in 1 minutes." read as a plan used up+ // ("usage limit") and went to the agent at once, never waited+ // for (zoecats on Discord). A limit that says it is back in+ // hours stays out of quota below+ return failRatecase status == 429 && rateWords.Match(body) && !plannedWords.Match(body):return failRatecase status >= 500 && quotaCodes.Match(body), status < 500 && usedUpWords.Match(body), status == 429 && plannedWords.Match(body):@@ -796,7 +806,7 @@ var resetsInWords = regexp.MustCompile(`(?i)\bresets? in (?:(\d+)\s*d(?:ays?)?\bvar quotaResetAfter = regexp.MustCompile(`(?i)\bquota will reset after ((?:\d+h)?(?:\d+m)?(?:\d+(?:\.\d+)?s)?)`)// resetsAfter is how long the refusal says until it is back-// (resetsInWords, quotaResetAfter), held to longestWritten. Zero when it+// (resetsInWords, quotaResetAfter, soonWords), held to longestWritten. Zero when it// doesn't say.func resetsAfter(body []byte) time.Duration {body = wordsOf(body)@@ -808,7 +818,52 @@ func resetsAfter(body []byte) time.Duration {return d}}- return 0+ return soonWritten(body)+}++// soonWords: a refusal that says in seconds, or in Chinese minutes, how+// long until it is back: "try again in 30 seconds", "retry after 20s",+// "resets in 45 secs", "请 1 分钟后重试", "30秒后再试". The English minutes,+// hours and days are resetsInWords' and tryAgainIn's.+var soonWords = regexp.MustCompile(`(?i)\b(?:resets? (?:in|after)|try again in|retry (?:in|after))\s*(\d+(?:\.\d+)?)\s*(?:s|secs?|seconds?)\b|(\d+)\s*(秒|分钟)钟?\s*(?:之?后)`)++// soonest is the longest a refusal may say it is back in for a 429 to be+// a rate limit waited out, not a plan used up: as long as a provider's+// patience can be set to wait (provider.PatienceLongest).+const soonest = time.Duration(provider.PatienceLongest) * time.Second++// soonWritten is the seconds, or Chinese minutes, a refusal says until it+// is back (soonWords). Zero when it doesn't say.+func soonWritten(body []byte) time.Duration {+ m := soonWords.FindSubmatch(body)+ switch {+ case m == nil:+ return 0+ case len(m[1]) > 0:+ n, _ := strconv.ParseFloat(string(m[1]), 64)+ return time.Duration(n * float64(time.Second))+ }+ n, _ := strconv.Atoi(string(m[2]))+ if string(m[3]) == "分钟" {+ return time.Duration(n) * time.Minute+ }+ return time.Duration(n) * time.Second+}++// backIn is how long a refusal says in words until it is back: resetsAfter,+// or a "try again in" (tryAgainIn). Zero when it doesn't say.+func backIn(body []byte) time.Duration {+ body = wordsOf(body)+ if d := resetsAfter(body); d > 0 {+ return d+ }+ return spanWritten(tryAgainIn.FindSubmatch(body))+}++// backSoon says a refusal says it is back within soonest.+func backSoon(body []byte) bool {+ d := backIn(body)+ return d > 0 && d <= soonest}// clineModelLimited says c, a provider whose requests reach Cline's API,diff --git a/internal/gateway/single_busy_test.go b/internal/gateway/single_busy_test.goindex 232840d2e..6d660e8d1 100644--- a/internal/gateway/single_busy_test.go+++ b/internal/gateway/single_busy_test.go@@ -160,6 +160,7 @@ func TestSingleProviderPassesALongLimitThrough(t *testing.T) {}{{"an hour", busyReply{code: 429, retryAfter: "3600", body: rateLimited}},{"quota", busyReply{code: 429, body: `{"type":"error","error":{"type":"rate_limit_error","message":"You exceeded your current quota, please check your plan and billing details"}}`}},+ {"usage limit back in hours", busyReply{code: 429, body: `{"type":"error","error":{"type":"rate_limit_error","message":"Usage has exceeded the limit. Your usage limit for the original model will reset in 5 hours."}}`}},{"key refused", busyReply{code: 401, body: `{"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"}}`}},} {t.Run(c.name, func(t *testing.T) {