Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

gateway: a plain 403 never rests an account until its window, and two accounts' alike 403 on one request rest neither (plugins#243)

yetoned5bc9f00092a
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index 292a113a4..22b16364d 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -94,7 +94,7 @@ recorded as a prediction failure (`noPrediction`).
- A rule by folder (`Rule.Folders`, `folders=~/work/acme,~/oss`; Cario0x on X: routing by the project being worked on) matches when the folder the agent works in is one of them or inside one (`provider.FolderUnder`, [`rulefolder.go`](../../internal/provider/rulefolder.go)): at a path's boundary, so `/a/b` is not under `/a/bc`; `~` is the home folder; a Windows drive is one with its WSL mount (`C:\work` is `/mnt/c/work`) and a `\\wsl$\<distro>` or `\\wsl.localhost\<distro>` share is the distro's own path; case doesn't count on macOS and Windows, nor ever for a drive. A rule outside folders (`Rule.ExceptFolders`, `except-folders=~/big-task`; #1606, Ti800) matches when the folder is in none of them, and an agent that says none isn't known to be in one, so it matches; a rule may have both (`Rule.ByFolder`, `inFolders`). The folder is read from the request only when a rule of the group has folders or except-folders (`workDir` in [`workdir.go`](../../internal/gateway/workdir.go)), in each agent's own words, taken from captured requests: Claude Code's ` - Primary working directory: <path>` (its `<`/`>` as entities; ` (was <old>)` after a change dropped), Codex's `<environment_context>` `<cwd>`, Pi's `<cwd>` in the system prompt, and OpenCode's `Working directory:` inside its `<env>` block. Only the system prompt and the user's and system messages' text count, never the model's words or a tool's result, and the last said wins. An agent that says none has no folder, and no `folders=` rule matches it (unknown, never empty). The same reading tells a plugin's fetch where its agent works ([provider-plugins](provider-plugins.md#where-the-agent-works-for-a-plugins-fetch)). A pause rule may hold by folder too (below). The trace's `RuleHit.Folder` is the folder read, and the Routing page tells it beside the rule's `folder …` condition. `TestFolderUnder`, `TestRuleMatchesByFolder`, `TestParseRuleFolders`, `TestWorkDirOfEachAgent`, `TestRuleByFolder`, `TestRuleMatchesExceptFolders`, `routing-folder.test.cjs`. Rules by folder are never in a shared group (`ShareGroup` drops every `ByFolder` rule).
- A pause rule (`Rule.Pause`, `pause=<model>`; John on Discord: a model of the group paused in its vendor's peak hours) leaves the model it names out of the group while it holds (`provider.PausedOut`, on `ruleClock`): the group's own rules and each nested group's on the way down to a member, a group named taking all its models, and a model named taking it wherever a group in the group has it (#625). It runs before planning, on every request rather than once a turn, so the model is not first, not a failover, and not kept by affinity, and comes back as soon as the hours end. It holds by hours, days, agents and folders only (`cleanRules` refuses another condition), and is not a send-to rule: `Matches`, `MatchRule`, `ThenUses`, `Ruled` and the compaction rules leave it out. A send-to rule naming a paused model finds it unready and the group routes as without it. The trace keeps what was paused (`GroupRef.Paused`), and the Routing page says so. Every model paused answers 503 (`pausedError`) naming each rule. Embeddings go by the same pauses; manual groups have no rules (`Live`), so none pause there. A pause by folder (#1606, Ti800: a subscription's quota kept for the long task in one folder, `pause=<model> except-folders=~/big-task`) reads the folder only when such a pause is on the way (`provider.PausesByFolder`, then `workDir`): outside its except-folders, which includes a request that says no folder (a subagent's or a small model's that doesn't), the model is left out; with `folders=`, only requests known to be in them. Embeddings and System One's decision calls have no folder. `Paused.Folder`/`ByFolder` carry what was read, and the Routing page says the folder or that none was said. `rules_pause_test.go`, `TestPauseByFolder`, `TestPauseByFolderRule`, `TestParsePauseByFolder`, `routing-pause-folder.test.cjs`.
3. Candidates come from `planGroup` for a group or `plan` for a provider. They are filtered by `sealedReaders` and `allowedCandidates`. An account past its share is held out of the candidates (`capHeld`): a usage cap's share, each window at its own where the account gives it one (`AccountWindowCaps`, willz on Discord: 50% of the five hours, 40% of the week) and at the account's cap otherwise, or 100% of a window for a Codex account set not to spend its credits (`provider.HoldCaps`, `WindowCaps.Of`, `CodexNoCredits`). A ChatGPT account's call on `codex-auto-review`, which OpenAI counts in no window, is held by no usage cap, only by credits it may not spend (`freeOnChatGPT`, #1593, `TestAccountCapLetsAutoReviewThrough`). The account is held while any window is past its own share, and `cappedError` and the trace name the fullest such window's share and cap. When every candidate is held and one held for its credits spends its resets by itself, `autoReset` spends a reset on it and it is tried (`planned.held`); otherwise a capped or barred set returns `cappedError` or `barredError`. Then `affine` moves the conversation's last answerer first, unless it is resting or nearly used up. An `X-Magpie-Account` header (`AccountHeader`) pins the request to one account: only that account is tried, and when it can't answer the caller is told why.
-4. The candidate loop calls `attempt` on each candidate in turn, through a `holdWriter`. The writer holds the reply until it is clearly passing. A candidate that fails before anything is sent rests (`failure`): out of credit for 30 minutes, out of quota until reset, rate limited until retry-after, otherwise a minute and longer on repeats: doubled each time, up to `longestRetry` (10 minutes) for a failure and `longestRateRest` (30 minutes) for a rate limit back as soon as its rest is over, or up to the provider's own `restCap` in minutes (`Provider.RestCapOr`, at most `MaxQuotaRest`; `magpie provider rest-cap <id> 4h`, #2008: a free key down for hours was asked every 10 minutes). A time the vendor names stands over it as before (`TestOwnRestCap`). The failure count is kept per key or account (`restKey`), and any of its models' answers clears it (`served`). Each model's own count (`restID`) is kept beside it and cleared only by that model's answer (`servedCandidate`): once it is past the account's, a sibling model answered since it last did, so the failure is the model's and it climbs its own ladder and rests alone, its siblings still served (#2241, weizsw: an Antigravity seat's gemini-3.5-flash-lite rested a minute 41 times of 43 while gemini-3.8-flash kept answering; `TestModelLadderClimbsPastSiblingAnswers`). Every model failing with no answer between still climbs the account's ladder and rests the account, and a candidate naming no model has the account's count alone. `rest-cap <id> off` (`provider.RestCapOff`, `restCap: -1`) is for a relay or account pool whose unknown errors come now and then and pass by the next request (Kismet Alan on Discord): a failure that says nothing of why (`failOther`: a 5xx, a dropped connection) then rests nobody and isn't counted, its trace rest `by: "off"` with no time left. A 401 or 403, a model not served, an account's window full, a rate limit (magpie's own 30-minute cap), quota, credit and a restriction still rest it as before (`TestRestCapOff`). The shortest cap otherwise is a minute, the first rest; `rest-cap <id> 30s` is turned away and names `off`. A 400, 403, 404 or 422 that says the model isn't served (`unservedWords`) and says nothing of quota, credit or a rate limit rests only that key's or account's model (`modelRefused`, keyed by `restID`, its backoff counted apart): its other models are still served first in order (#1235, SenseNova's 403 "model is not available in the current token plan"). A Google account's Code Assist (the Antigravity and Gemini CLI built-ins) answering 404 `NOT_FOUND` ("Requested entity was not found.") rests its model the same way, as its endpoint is the same for every model (`codeAssistModelMissing`; Packing1 on Discord: Argon, listed for a Google AI Pro account, answered 404 and rested the whole account; `TestCodeAssistMissingModelRestsTheModel`). A refusal for want of quota, credit or rate, for an account its vendor restricted, or for a verify it asks of its account there, from another computer's magpie (a Remote magpie provider, LAN share) rests only the model it was about (`remoteModelRest`, keyed by `restID`, in `restAfterMarked` and `rateRest`): that magpie serves each of its models from its own providers and rests them itself, so one out of quota there says nothing of the others (bigsteven on Discord: a 429 for one model rested every model on the remote for the 3 hours it named, so a group never moved on to the remote's other models; `TestRemoteMagpieQuotaRestsTheModel`, `TestRemoteMagpieRestrictedRestsTheModel`). Its key refused, the computer unreachable, or the key's own limit there (`keyLimited`'s 429, which carries `X-Magpie-Limit-Reset`, `remoteKeyLimitHeader`), still rests it whole: #2077's reporter had a Claude Code session limit on the other magpie refuse its Qoder models here (`TestRemoteMagpieRestsTheModel`, two gateways over HTTP; `TestRemoteMagpieKeyLimitRestsTheKey`). A 400, 404, 410 or 422 that says the model is retired (`retiredWords`: OpenCode Zen's 410 `ModelDeprecated` "Model exo-free has been deprecated.", whose own model list still names it; MOMO on Discord) rests only that model the same way, is retryable so a group's next member or another provider of it is asked, and is remembered (`provider.Retire`, `retired.json` beside `providers.json`): the model leaves that provider's entries in the catalog (`buildEntries`), so /v1/models, every agent's model list and the found groups go without it, until a request on it is answered again (`Unretire` in `servedCandidate`) or a week has passed. A refresh of the vendor's list doesn't bring it back early, since that list is where it still is. A `provider/model` request for it is still sent. A 400 or 422 that says this upstream's API can't take the request's shape (`shapeWords`: xAI's 422 "unknown item type" (#350), OpenAI's "Unknown parameter", and a limit of a model's own on tool schemas, OpenCode Go's 400 "JSON schema exceeds the maximum nesting depth of 10 levels" on muse-spark-1.3-contributor, #1870), or that it is served on another API than the one asked (`wrongEndpoint`, once the member has tried every API it has a URL for: #2670, a relay with a Chat URL alone refusing tools with reasoning and `none` too), rests nobody: the next candidate is asked once (`shapeRefused`, `failShape`), and the last one's error reaches the agent. A request missing what every API requires ("field required") isn't one, nor is a 200 that mentions it. Command Code's Provider API turning a request away with nothing but a trace id (`unsaidWords`: `{"message":"invalid request error trace_id: <hex>","type":"invalid_request_error"}`, a 400, or a 422 before) is handled the same way: it turns away a request whose history and asked-for output together are over a limit below the model's listed context (the reporter's A/B: one ~748K-token agent conversation answered 200 with `max_completion_tokens` 64000 and this 400 with 384000, the model's listed max output), which another member's model may take, and the key is fine (#2313, `TestGroupFailsOverCommandCodesBare400`). Its overflow, which names the model's context length before the trace id, and an error that names a `param`, aren't one. A relay's own cap on one request's input that says so (`tooLongRe`: B.ai's 400 "Input token exceed the limit", on a 380K request under the model's 1M window; #2445, worksyang) is taken as an overflow (`overflowed`), which no word list of `retryable` matched: a group's next member that may hold the request is asked (`withRoom`, `failOverflow`), nobody rests, another account of the same model isn't asked, and with no member left the agent is told the conversation is too long, so it compacts. A rate limit worded with input tokens (`paceWords`) isn't one, and a plain bad request is still asked of nobody else (`TestRelayInputCapGoesToTheNextMember`). A subscription account that fails any other way with a window full, by magpie's latest reading of its windows, rests until that window renews. Both quota rests and window rests are held to `longestQuota` (8 days), so a reset time read wrong (a year off) doesn't bench the account for that long. The next candidate is then tried. A failed provider waits `fallbackCooldown` (1 minute) at the back. A WorkBuddy account the vendor restricted (#1849: 403 `{"code":11140,"msg":"request illegal","displayMsg":{"en":"Your account has been restricted due to unusual activity",…}}`) rests `restrictedRest` (2 hours) as `restricted` (`failRestricted`), not the minutes of an unknown failure: only WorkBuddy lifting the restriction, after the user's appeal, brings it back. Both shapes are known: the built-in's `wbExplain` and the plugin's `explained()` (0.1.17, the mover's `min`) end the error with `provider.WBRestrictedHint` (`restrictedSaid`), and the vendor's own body, its code 11140 or its words (`provider.WBRestricted`), counts only from a WorkBuddy candidate (`restrictedRefused`, `isWorkBuddy`), as another vendor's 11140 is its own. Bare "request illegal" isn't matched. WorkBuddy's 11128 channel refusal stays `promptRefused` and rests nobody. The Routing page names it "account restricted by the vendor", says the appeal is the way back, and gives the account's row a button that lifts the rest (`/api/gateway/unrest`) once the vendor has. Every other rest a key or account takes (quota, rate limit, retry-after, a failure's backoff) can be lifted by hand the same way (#1869): "Try now" on its row on the Routing page and beside its rest note in the provider's key list. It ends the rest only: the backoff count stays, and a refusal after it rests the key again the usual way. The vendor's safety filter refusing a reply rests nobody, whether it refuses the turn before any of the reply is said or once some of it has been said (`refused`, `refusedAfter`). Its note on the try (and in the error the agent gets when nobody is left) is the vendor's reason, with Anthropic's `stop_details` category and explanation when it gave them (`refusedDetail`, #1768: "safety filter: refusal (category cyber)"); none is made up when it gave none. `stop_details` is carried through the IR (`Event.StopDetails`), so a Claude subscription run through the claude CLI and a translated Anthropic route keep it for an Anthropic client too. How many more candidates a refusal before any of the reply is asked of is Settings' `RefusalRetries` (`refusalAsksAgain`, counted per request in the candidate loop; a group's other members count too): unset, every one left (#248); 0, none (#1925, DD on Discord). When nobody is left, an agent on Anthropic's Messages API gets the held refusal itself, the vendor's 200 with `stop_reason` `refusal` and its `stop_details`, streamed or whole (`holdWriter.handBack`), since that is what Claude Code switches to its fallback model on; a 400 is only an API error to it. Codex and Chat clients still get `failTo`'s 400 (`refusedStatus`), which they don't ask again for. `TestRefusalRetries*` in [`refusal_retries_test.go`](../../internal/gateway/refusal_retries_test.go) cover keys, a group and the Claude subscription. The ChatGPT backend's 502 "response protection is unavailable" is about the request's content, not the account: the same history fails the same on each of its accounts and models. It rests nobody, isn't tried again after a moment, and isn't asked of the provider's other accounts or models; another provider left in a group is asked once (`protectionRefused`, `elsewhere`). One cause is known (#1270): a history that replays the backend's own `web_search_call` in a request that doesn't declare the `web_search` tool is refused so, whatever the account, model or turn (a plain turn, a thread's description, a compaction, Codex's own text compaction for a custom provider with `tools: []`). Every request to the ChatGPT backend with such a history therefore declares cached-only search (`{"type":"web_search","external_web_access":false}`, `provider.DeclareSearch`): an account's in `codexBody`, Codex's own sign-in relayed in `codexUpstreamOn`. It goes in the tools, or for Responses Lite (its header, `reasoning.context` `all_turns`, or an `additional_tools` item already there), which refuses a top-level `web_search`, in the `additional_tools` item. A request that offered no tools gets `tool_choice` `none` so the model can't search; one with a forced choice keeps it. A history without a search, or a request that declares search already, is relayed unchanged. siwc, API-key requests to openai.com and the legacy `/responses/compact` are left as they were.
+4. The candidate loop calls `attempt` on each candidate in turn, through a `holdWriter`. The writer holds the reply until it is clearly passing. A candidate that fails before anything is sent rests (`failure`): out of credit for 30 minutes, out of quota until reset, rate limited until retry-after, otherwise a minute and longer on repeats: doubled each time, up to `longestRetry` (10 minutes) for a failure and `longestRateRest` (30 minutes) for a rate limit back as soon as its rest is over, or up to the provider's own `restCap` in minutes (`Provider.RestCapOr`, at most `MaxQuotaRest`; `magpie provider rest-cap <id> 4h`, #2008: a free key down for hours was asked every 10 minutes). A time the vendor names stands over it as before (`TestOwnRestCap`). The failure count is kept per key or account (`restKey`), and any of its models' answers clears it (`served`). Each model's own count (`restID`) is kept beside it and cleared only by that model's answer (`servedCandidate`): once it is past the account's, a sibling model answered since it last did, so the failure is the model's and it climbs its own ladder and rests alone, its siblings still served (#2241, weizsw: an Antigravity seat's gemini-3.5-flash-lite rested a minute 41 times of 43 while gemini-3.8-flash kept answering; `TestModelLadderClimbsPastSiblingAnswers`). Every model failing with no answer between still climbs the account's ladder and rests the account, and a candidate naming no model has the account's count alone. `rest-cap <id> off` (`provider.RestCapOff`, `restCap: -1`) is for a relay or account pool whose unknown errors come now and then and pass by the next request (Kismet Alan on Discord): a failure that says nothing of why (`failOther`: a 5xx, a dropped connection) then rests nobody and isn't counted, its trace rest `by: "off"` with no time left. A 401 or 403, a model not served, an account's window full, a rate limit (magpie's own 30-minute cap), quota, credit and a restriction still rest it as before (`TestRestCapOff`). The shortest cap otherwise is a minute, the first rest; `rest-cap <id> 30s` is turned away and names `off`. A 400, 403, 404 or 422 that says the model isn't served (`unservedWords`) and says nothing of quota, credit or a rate limit rests only that key's or account's model (`modelRefused`, keyed by `restID`, its backoff counted apart): its other models are still served first in order (#1235, SenseNova's 403 "model is not available in the current token plan"). A Google account's Code Assist (the Antigravity and Gemini CLI built-ins) answering 404 `NOT_FOUND` ("Requested entity was not found.") rests its model the same way, as its endpoint is the same for every model (`codeAssistModelMissing`; Packing1 on Discord: Argon, listed for a Google AI Pro account, answered 404 and rested the whole account; `TestCodeAssistMissingModelRestsTheModel`). A refusal for want of quota, credit or rate, for an account its vendor restricted, or for a verify it asks of its account there, from another computer's magpie (a Remote magpie provider, LAN share) rests only the model it was about (`remoteModelRest`, keyed by `restID`, in `restAfterMarked` and `rateRest`): that magpie serves each of its models from its own providers and rests them itself, so one out of quota there says nothing of the others (bigsteven on Discord: a 429 for one model rested every model on the remote for the 3 hours it named, so a group never moved on to the remote's other models; `TestRemoteMagpieQuotaRestsTheModel`, `TestRemoteMagpieRestrictedRestsTheModel`). Its key refused, the computer unreachable, or the key's own limit there (`keyLimited`'s 429, which carries `X-Magpie-Limit-Reset`, `remoteKeyLimitHeader`), still rests it whole: #2077's reporter had a Claude Code session limit on the other magpie refuse its Qoder models here (`TestRemoteMagpieRestsTheModel`, two gateways over HTTP; `TestRemoteMagpieKeyLimitRestsTheKey`). A 400, 404, 410 or 422 that says the model is retired (`retiredWords`: OpenCode Zen's 410 `ModelDeprecated` "Model exo-free has been deprecated.", whose own model list still names it; MOMO on Discord) rests only that model the same way, is retryable so a group's next member or another provider of it is asked, and is remembered (`provider.Retire`, `retired.json` beside `providers.json`): the model leaves that provider's entries in the catalog (`buildEntries`), so /v1/models, every agent's model list and the found groups go without it, until a request on it is answered again (`Unretire` in `servedCandidate`) or a week has passed. A refresh of the vendor's list doesn't bring it back early, since that list is where it still is. A `provider/model` request for it is still sent. A 400 or 422 that says this upstream's API can't take the request's shape (`shapeWords`: xAI's 422 "unknown item type" (#350), OpenAI's "Unknown parameter", and a limit of a model's own on tool schemas, OpenCode Go's 400 "JSON schema exceeds the maximum nesting depth of 10 levels" on muse-spark-1.3-contributor, #1870), or that it is served on another API than the one asked (`wrongEndpoint`, once the member has tried every API it has a URL for: #2670, a relay with a Chat URL alone refusing tools with reasoning and `none` too), rests nobody: the next candidate is asked once (`shapeRefused`, `failShape`), and the last one's error reaches the agent. A request missing what every API requires ("field required") isn't one, nor is a 200 that mentions it. Command Code's Provider API turning a request away with nothing but a trace id (`unsaidWords`: `{"message":"invalid request error trace_id: <hex>","type":"invalid_request_error"}`, a 400, or a 422 before) is handled the same way: it turns away a request whose history and asked-for output together are over a limit below the model's listed context (the reporter's A/B: one ~748K-token agent conversation answered 200 with `max_completion_tokens` 64000 and this 400 with 384000, the model's listed max output), which another member's model may take, and the key is fine (#2313, `TestGroupFailsOverCommandCodesBare400`). Its overflow, which names the model's context length before the trace id, and an error that names a `param`, aren't one. A relay's own cap on one request's input that says so (`tooLongRe`: B.ai's 400 "Input token exceed the limit", on a 380K request under the model's 1M window; #2445, worksyang) is taken as an overflow (`overflowed`), which no word list of `retryable` matched: a group's next member that may hold the request is asked (`withRoom`, `failOverflow`), nobody rests, another account of the same model isn't asked, and with no member left the agent is told the conversation is too long, so it compacts. A rate limit worded with input tokens (`paceWords`) isn't one, and a plain bad request is still asked of nobody else (`TestRelayInputCapGoesToTheNextMember`). A subscription account that fails any other way with a window full, by magpie's latest reading of its windows, rests until that window renews. A 401 or 403 is the exception: one that names a quota is out of quota already (`usedUpWords`), and one that doesn't is about the key or the request, so it rests on the backoff ladder whatever the reading says (plugins#243, zygs1083-dotcom: a Factory account read at 98% sat out its week over one plain 403; `TestPlain403DoesNotRestUntilTheWindow`). A plain 403 (`plainForbidden`: no quota, credit, rate, verification, restriction or model not served) from a second key or account of the same provider, model and reasoning in one request is the request refused, not each account: neither rests, the first one's rest and counted failure are taken back (`forbiddenAlike`, `unforbid`), and the provider's other mates are asked last (`matesLast`). Before, that one request walked every account of the provider into backoff (`TestPlain403OnTwoMatesRestsNeither`). One account's plain 403 alone still rests it. Both quota rests and window rests are held to `longestQuota` (8 days), so a reset time read wrong (a year off) doesn't bench the account for that long. The next candidate is then tried. A failed provider waits `fallbackCooldown` (1 minute) at the back. A WorkBuddy account the vendor restricted (#1849: 403 `{"code":11140,"msg":"request illegal","displayMsg":{"en":"Your account has been restricted due to unusual activity",…}}`) rests `restrictedRest` (2 hours) as `restricted` (`failRestricted`), not the minutes of an unknown failure: only WorkBuddy lifting the restriction, after the user's appeal, brings it back. Both shapes are known: the built-in's `wbExplain` and the plugin's `explained()` (0.1.17, the mover's `min`) end the error with `provider.WBRestrictedHint` (`restrictedSaid`), and the vendor's own body, its code 11140 or its words (`provider.WBRestricted`), counts only from a WorkBuddy candidate (`restrictedRefused`, `isWorkBuddy`), as another vendor's 11140 is its own. Bare "request illegal" isn't matched. WorkBuddy's 11128 channel refusal stays `promptRefused` and rests nobody. The Routing page names it "account restricted by the vendor", says the appeal is the way back, and gives the account's row a button that lifts the rest (`/api/gateway/unrest`) once the vendor has. Every other rest a key or account takes (quota, rate limit, retry-after, a failure's backoff) can be lifted by hand the same way (#1869): "Try now" on its row on the Routing page and beside its rest note in the provider's key list. It ends the rest only: the backoff count stays, and a refusal after it rests the key again the usual way. The vendor's safety filter refusing a reply rests nobody, whether it refuses the turn before any of the reply is said or once some of it has been said (`refused`, `refusedAfter`). Its note on the try (and in the error the agent gets when nobody is left) is the vendor's reason, with Anthropic's `stop_details` category and explanation when it gave them (`refusedDetail`, #1768: "safety filter: refusal (category cyber)"); none is made up when it gave none. `stop_details` is carried through the IR (`Event.StopDetails`), so a Claude subscription run through the claude CLI and a translated Anthropic route keep it for an Anthropic client too. How many more candidates a refusal before any of the reply is asked of is Settings' `RefusalRetries` (`refusalAsksAgain`, counted per request in the candidate loop; a group's other members count too): unset, every one left (#248); 0, none (#1925, DD on Discord). When nobody is left, an agent on Anthropic's Messages API gets the held refusal itself, the vendor's 200 with `stop_reason` `refusal` and its `stop_details`, streamed or whole (`holdWriter.handBack`), since that is what Claude Code switches to its fallback model on; a 400 is only an API error to it. Codex and Chat clients still get `failTo`'s 400 (`refusedStatus`), which they don't ask again for. `TestRefusalRetries*` in [`refusal_retries_test.go`](../../internal/gateway/refusal_retries_test.go) cover keys, a group and the Claude subscription. The ChatGPT backend's 502 "response protection is unavailable" is about the request's content, not the account: the same history fails the same on each of its accounts and models. It rests nobody, isn't tried again after a moment, and isn't asked of the provider's other accounts or models; another provider left in a group is asked once (`protectionRefused`, `elsewhere`). One cause is known (#1270): a history that replays the backend's own `web_search_call` in a request that doesn't declare the `web_search` tool is refused so, whatever the account, model or turn (a plain turn, a thread's description, a compaction, Codex's own text compaction for a custom provider with `tools: []`). Every request to the ChatGPT backend with such a history therefore declares cached-only search (`{"type":"web_search","external_web_access":false}`, `provider.DeclareSearch`): an account's in `codexBody`, Codex's own sign-in relayed in `codexUpstreamOn`. It goes in the tools, or for Responses Lite (its header, `reasoning.context` `all_turns`, or an `additional_tools` item already there), which refuses a top-level `web_search`, in the `additional_tools` item. A request that offered no tools gets `tool_choice` `none` so the model can't search; one with a forced choice keeps it. A history without a search, or a request that declares search already, is relayed unchanged. siwc, API-key requests to openai.com and the legacy `/responses/compact` are left as they were.
5. The last candidate's error reaches the agent, with one exception for a group (#1418, `patience.go`). When every member has failed in a way that passes (`busy`: 408, 500, 502, 503, 504 or 529 that isn't a proxy, a quota, response protection, or an upstream that answered a stream request with a whole reply, which fails the same however often it is asked (`didNotStream`; the same-one retries of `passing` leave it alone too, `TestStreamlessMemberIsAskedOnce`), or a 429 that is a short rate limit, not a plan or credit), and nothing of the reply has been sent, the members that failed so are asked again in turn after a pause (`replan`). The pause is `retryPause` doubled each round up to `replanLongest()` (30 times `retryPause`, 30s), and never shorter than the soonest Retry-After; a member whose Retry-After ends past the pause is asked after the others. This goes on while the group's `Patience` lasts. A member of a group in the group uses the innermost that sets `Patience` (`innerPatience`, #2410), as FirstToken (#2246) and Sink (#2255) do; unset falls back to the group asked for, and an inner `PatienceOff` skips replan for that member (`TestNestedGroupPatienceThroughOuter`, `TestNestedGroupPatienceInnerOff`, `TestNestedGroupPatienceFallsBackToOuter`). `replanRounds` (12) is only a backstop that never comes first: the pauses alone add up to `PatienceLongest` in 11 rounds (with 6, a group set to 240s gave up after 90s of pauses, real-jacket on #1418; `TestPatienceIsUsedUp`). The budget is the group's `Patience`, counted from the first time every member had failed, when the agent would have had the error (`waitFrom`; `Group.Waits`: 0 is `PatienceDefault`, 60s; -1, `PatienceOff`, turns it off; at most `PatienceLongest`, 240s). A pause that would end past the budget isn't taken, and the agent gets the error. While another member is busy too, the last one isn't first tried again alone (`passing`): the members take turns, and once the budget has begun, the same-one retries aren't taken either. While the budget lasts, the last candidate's failure is held (`newHoldWriter`), and a streaming agent is kept alive through the pause with SSE comments (`pauseAlive`). The try is traced with `Again` (the pause), `Replan` (how many are asked again) and `Patience` (the budget in seconds). The group editor's "Every member busy" row sets it. A plain provider keeps only the same-one retries `passing` gives the last one left. An agent that goes away in either pause (the same-one retry's or a group's replan) ends the request there with 499: nobody is asked again, and the try whose pause it was is marked `Gone` (#1894, heng91: a relay down while its proxy was off, the agent's window closed). The Routing page says the agent canceled the request before the retry and lists it as canceled, rather than ending its story on "tried again in 2s", a retry that never came (`TestAgentGoneInTheRetryPause`, `routing-agent-gone.test.cjs`). A try that answered is recorded (`served`, `answered`) for routing, affinity and usage. A try is added to the route's `Tries` only as it is asked, so a later try is the record that the gateway went on from this one, and the Routing page tells what came of a try from that, never from where it stood in the order (#2281, xishuchkui-code: an Antigravity 502 empty reply that went on to the next account was told as "an error another account wouldn't fix, so Claude Code gets it"). The one case the record can't show is the last try left out of its allowance after an earlier failure that wasn't one, where the agent is told the earlier error: that try is marked `Earlier` (`routing-went-on.test.cjs`).
## Constraints and failure behavior
@@ -715,6 +715,7 @@ the row of the entry that id reaches, with its label, under the bare id.
```sh
go test -tags nogui ./internal/gateway -run 'TestGroupAsksOneRestingMemberNotEach|TestLoneRestingMemberStillAsked|TestRetrievalGroupAsksOneRestingMember|TestProbeOne|TestLoop|TestLongReplyThatDoesNotLoop|TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestModelNotInPlanRestsTheModelNotTheKey|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestWindowCapsHoldTheAccount|TestAccountCap|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestSearchFollowsTheSearchersProxy|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestProtection|TestSearchedConversation|TestUnsearchedConversationUnchanged|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestClaudeAccountBackOnceItsWindowRenews|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestCodexHeardWithinItsIdleTimeout|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused|TestAIStudioThinking|TestGeminiThoughtsThroughLocalProxy|TestGeminiRefusedLevel|TestGeminiThinkingOffAtItsLowest|TestThinkingOff|TestTranslatedEffortLoggedAsSent|TestTranslatedThinkingOffRefused|TestSub2APIThinkingOffRefused|TestRefusesThinkingOff|TestClaudeFiveFiveThinkingOffOnTheWire|TestSamplingAsTheModelTakesIt|TestHaikuFiveFiveSamplingOnTheWire|TestGeminiThoughtsMarkedInExtraContent|TestGeminiThinkingLevelRefused|TestGeminiMinimalRefusedTwiceAtOnce|TestPassthroughGeminiMinimalRefused|TestPassthroughEffortNoneRefused|TestPassthroughMinimalRefusedNoneTaken|TestOtherOffNamed|TestClaudeSignInNotToldItsSignInFailed|TestSignInWriterReleasesA403|TestSignInBetaStaysWithMagpie|TestStructuredOutput|TestResponsesTextKeepsVerbosityBesideFormat|TestRefusedFormatToldInWords|RPM|TestQueued|TestCustomGeminiUpstream|TestWholeNums|TestGrokCalls|TestOnlyGrokCalls|TestGrokModel|TestGrokPluginCalls'
go test -tags nogui ./internal/gateway -run 'TestMemberTooSmallIsNotAsked|TestMemberTooSmallIsTheLastResort|TestClaudeAccountsSpentFallBackToCodex|TestConversationOutgrowingItsMemberMoves|TestRoomyLeavesWhatItCantJudge|TestPayloadTooLarge|TestOverflow|TestContextOverflow|TestRelayInputCap|TestEveryVendorsOverflow'
+go test -tags nogui ./internal/gateway -run 'TestPlain403|TestOrderedKeepsAnAccountAt98|TestPromptRefusalRestsNobody'
go test -tags nogui ./internal/gateway -run 'TestVertex'
go test -tags nogui ./internal/gateway -run 'TestBedrock'
go test -tags nogui ./internal/gateway -run 'TestPublicAddr|TestEditImageURL|TestPrivateDialer|TestFetchPrivateEntries|TestVendorImageURLAtPrivateAddress|TestVideoContentAtPrivateAddress'
diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go
index 2f1f73030..3619bbfca 100644
--- a/internal/gateway/gateway.go
+++ b/internal/gateway/gateway.go
@@ -2253,6 +2253,7 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
// the key or subscription account the last try went to (#557)
providerKeyID, providerKeyName, providerAccount := "", "", ""
var other *Try // the first failure that wasn't an allowance run out
+ var forbade []forbidden // the plain 403s rested so far (forbiddenAlike)
autoReset := resetFirst != nil // a Codex or Claude reset looked at, once a request
// what the tries' held streams sent the agent ahead of a reply (#751)
kept := &keptAlive{proto: from, told: start}
@@ -3169,13 +3170,47 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
matesLast(cands[i+1:], c)
continue
}
+ plain403 := hw.failed() && !hw.refused && plainForbidden(c, hw.code(), hw.errBody())
+ if alike, others := forbiddenAlike(forbade, c); plain403 && len(alike) > 0 {
+ // a mate refused this request with a plain 403 too: the
+ // request is what is refused, not each account (plugins#243).
+ // Neither rests, the mate's rest taken back, and its other
+ // mates are asked last, as for a prompt turned away
+ for _, f := range alike {
+ unforbid(f)
+ }
+ forbade = others // each taken back once
+ try.Fail = failOther
+ s.trace.update(tr, func(t *Route) {
+ for _, f := range alike {
+ if f.try >= 0 && f.try < len(t.Tries) {
+ t.Tries[f.try].Rest = nil
+ }
+ }
+ if !last {
+ t.Tries[len(t.Tries)-1] = try
+ }
+ })
+ if !last {
+ if other == nil {
+ other = &Try{Status: call.Status, Error: call.Error}
+ }
+ skipped = append(skipped, c.label()+": "+call.Error)
+ matesLast(cands[i+1:], c)
+ continue
+ }
+ }
if !last && hw.failed() {
if f := failure(hw.code(), hw.errBody()); other == nil && f != failQuota && f != failCredit {
other = &Try{Status: call.Status, Error: call.Error}
}
rest := s.restAfterMarked(c, hw.code(), hw.header, hw.errBody(), hw.sharedPool)
try.Fail, try.Rest = rest.Why, &rest
- s.trace.update(tr, func(t *Route) { t.Tries[len(t.Tries)-1] = try })
+ at := -1
+ s.trace.update(tr, func(t *Route) { t.Tries[len(t.Tries)-1], at = try, len(t.Tries)-1 })
+ if plain403 {
+ forbade = append(forbade, forbidden{c: c, rest: rest, try: at})
+ }
skipped = append(skipped, c.label()+": "+call.Error)
if waitsOf(c.patience, patienceFallback) > 0 && !hw.turnedAway && busy(hw.code(), hw.errBody()) {
busyNow = noteBusy(busyNow, c, hw.header)
diff --git a/internal/gateway/plain403_rest_test.go b/internal/gateway/plain403_rest_test.go
new file mode 100644
index 000000000..301dcc369
--- /dev/null
+++ b/internal/gateway/plain403_rest_test.go
@@ -0,0 +1,124 @@
+package gateway
+
+import (
+ "io"
+ "net/http"
+ "slices"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/yetone/magpie/internal/provider"
+)
+
+// A plain 403 (or 401) that names no quota is about the key or the
+// request, not the window: an account whose cached usage read 98% rests
+// on the backoff ladder, not until its window resets. Before, it sat out
+// the whole week on the reading alone (plugins#243, zygs1083-dotcom: one
+// Factory 403 benched a working account into its reset). A 403 that does
+// say the quota is spent still rests until the window, and so does any
+// other failure at 98%, as before.
+func TestPlain403DoesNotRestUntilTheWindow(t *testing.T) {
+ forgetRouting()
+ old := allowances
+ defer func() { allowances = old }()
+ now := time.Now()
+ week := now.Add(39 * time.Hour)
+ allowances = func(string) map[string]provider.Allowance {
+ return map[string]provider.Allowance{
+ "[email protected]": {{Used: 98, Resets: week, Span: 7 * 24 * time.Hour}},
+ }
+ }
+ acct := candidate{p: provider.Provider{ID: "factory", Account: &provider.Account{Agent: "factory", User: "[email protected]"}}, model: "claude-opus-4-7", rest: "factory#[email protected]"}
+ if !acct.full(now).Equal(week) {
+ t.Fatal("the account at 98% doesn't read as full; the test proves nothing")
+ }
+ s := &Server{}
+ near := func(d, want time.Duration) bool { return d > want-time.Minute && d <= want }
+
+ for _, status := range []int{http.StatusForbidden, http.StatusUnauthorized} {
+ forgetRouting()
+ r := s.restAfter(acct, status, http.Header{}, []byte(`{"error":{"message":"Forbidden"}}`))
+ if r.By == "window" || r.By != "backoff" || time.Until(r.Until) > longestRetry {
+ t.Fatalf("a plain %d at 98%% rests %v by %s (%s)", status, time.Until(r.Until).Round(time.Minute), r.By, r.Why)
+ }
+ }
+
+ // a 403 that says the quota is spent is out until the window
+ forgetRouting()
+ r := s.restAfter(acct, http.StatusForbidden, http.Header{}, []byte(`{"error":{"message":"You've hit your usage limit"}}`))
+ if r.Why != failQuota || r.By != "window" || !near(time.Until(r.Until), 39*time.Hour) {
+ t.Fatalf("a 403 saying used up rests %v by %s (%s)", time.Until(r.Until).Round(time.Minute), r.By, r.Why)
+ }
+ // and another failure at 98% still trusts the reading, as before
+ forgetRouting()
+ r = s.restAfter(acct, http.StatusBadGateway, http.Header{}, []byte("bad gateway"))
+ if r.By != "window" || !near(time.Until(r.Until), 39*time.Hour) {
+ t.Fatalf("a 502 at 98%% rests %v by %s", time.Until(r.Until).Round(time.Minute), r.By)
+ }
+}
+
+// One request refused with a plain 403 by two accounts of a provider is
+// the request refused, not each account (plugins#243, zygs1083-dotcom:
+// Factory's 403 for a Claude Code wrapper it refused walked every account
+// into backoff, so the next requests found them all resting). Neither
+// rests, and an account after them answers. One account's plain 403
+// alone still rests it, as a key refused does.
+func TestPlain403OnTwoMatesRestsNeither(t *testing.T) {
+ forbid := func(w http.ResponseWriter) {
+ w.Header().Set("Content-Type", "application/json")
+ w.WriteHeader(http.StatusForbidden)
+ io.WriteString(w, `{"error":{"message":"Forbidden"}}`)
+ }
+ for _, x := range []struct {
+ name string
+ spares []string
+ also []string // accounts after acct-1 that refuse too
+ resting []string // who rests after
+ code int
+ }{
+ {"alone", []string{"[email protected]"}, nil, []string{"codex@[email protected]"}, 200},
+ {"two alike, a third answers", []string{"[email protected]", "[email protected]"}, []string{"acct-2"}, nil, 200},
+ {"two alike, nobody left", []string{"[email protected]"}, []string{"acct-2"}, nil, 403},
+ } {
+ t.Run(x.name, func(t *testing.T) {
+ codexSignedIn(t, x.spares...)
+ if err := provider.SetRouting("codex", provider.Ordered); err != nil {
+ t.Fatal(err)
+ }
+ var tried []string
+ chatgptRefusing(t, &tried, forbid, x.also...)
+ s := New()
+ code, body := codexAskOn(s, `{"model":"gpt-5.5","stream":true,"input":"ping"}`)
+ if code != x.code {
+ t.Fatalf("status %d: %s; tried %v", code, body, tried)
+ }
+ r := lastRoute(s)
+ for _, who := range append([]string{"codex@[email protected]"}, func() (w []string) {
+ for _, s := range x.spares {
+ w = append(w, "codex@"+s)
+ }
+ return
+ }()...) {
+ _, ok := restOf(who)
+ if want := slices.Contains(x.resting, who); ok != want {
+ t.Fatalf("%s resting %v, want %v; tried %v; tries %+v", who, ok, want, tried, r.Tries)
+ }
+ if !slices.Contains(x.resting, who) {
+ routed.Lock()
+ n := routed.failures[who]
+ routed.Unlock()
+ if n != 0 {
+ t.Fatalf("%s still counts %d failures", who, n)
+ }
+ }
+ }
+ if want := map[int]string{200: "acct-1,acct-2", 403: "acct-1,acct-2"}[code]; x.also != nil && !strings.HasPrefix(strings.Join(tried, ","), want) {
+ t.Fatalf("tried %v", tried)
+ }
+ if x.also != nil && (r.Tries[0].Rest != nil || r.Tries[1].Rest != nil) {
+ t.Fatalf("tries still say they rest: %+v", r.Tries)
+ }
+ })
+ }
+}
diff --git a/internal/gateway/routing.go b/internal/gateway/routing.go
index 69a84685b..d9b81ec87 100644
--- a/internal/gateway/routing.go
+++ b/internal/gateway/routing.go
@@ -79,6 +79,68 @@ func servedCandidate(c candidate, tokens int) {
}
}
+// forbidden is a plain 403 one request was given, and the rest it took:
+// what forbiddenAlike and unforbid read.
+type forbidden struct {
+ c candidate
+ rest Rest
+ try int // its index in the route's Tries, -1 untraced
+}
+
+// plainForbidden says a reply is a 403 that says nothing of why: no
+// quota, credit, rate, verification, restriction, a model not served.
+// Such a 403 may be the key's or the request's.
+func plainForbidden(c candidate, status int, body []byte) bool {
+ return status == http.StatusForbidden && failureOf(c, status, body) == failOther &&
+ !modelRefused(status, body) && !codeAssistModelMissing(c.p, status, body)
+}
+
+// forbiddenAlike is the plain 403s of c's mates earlier in the request:
+// another key or account of its provider, asked the same model at the
+// same reasoning. Two of them refusing one request alike, it is the
+// request that is refused, not each account (plugins#243,
+// zygs1083-dotcom: Factory's 403 for a Claude Code wrapper it refused
+// rested every account in turn, and the next requests found them all in
+// backoff).
+// The rest of seen is returned beside them, for the request to keep.
+func forbiddenAlike(seen []forbidden, c candidate) (alike, others []forbidden) {
+ for _, f := range seen {
+ if f.c.p.ID == c.p.ID && f.c.model == c.model && f.c.effort == c.effort && f.c.who() != c.who() {
+ alike = append(alike, f)
+ } else {
+ others = append(others, f)
+ }
+ }
+ return alike, others
+}
+
+// unforbid takes back the rest a plain 403 took and the failures it
+// counted, when a mate's alike 403 said the request was what was refused.
+// A rest taken since, which isn't this one, stays.
+func unforbid(f forbidden) {
+ if f.rest.By == "off" || f.rest.Key == "" {
+ return // it neither rested nor counted
+ }
+ routed.Lock()
+ for _, k := range []string{f.c.restKey(), f.c.restID()} {
+ if n := routed.failures[k]; n > 1 {
+ routed.failures[k] = n - 1
+ } else {
+ delete(routed.failures, k)
+ }
+ if f.c.model == "" {
+ break // restID is restKey: counted once
+ }
+ }
+ routed.Unlock()
+ restingUntil.Lock()
+ if until, ok := restingUntil.m[f.rest.Key]; ok && until.Equal(f.rest.Until) {
+ delete(restingUntil.m, f.rest.Key)
+ delete(restingUntil.note, f.rest.Key)
+ }
+ restingUntil.Unlock()
+}
+
// clearRest wakes a candidate that just answered, whether it was tried after
// a provider rest or after a model-specific rest.
func clearRest(key string) {
@@ -865,8 +927,12 @@ func (s *Server) restAfterMarked(c candidate, status int, header http.Header, bo
// everyone (sub2api's 503) says nothing of the key's own windows.
// Held to the longest an account out of quota sits out: the window
// is the vendor's reading, and one read a year off (a /usage date
- // with no year) would bench the account until someone noticed
- if t := c.full(now); !t.IsZero() && c.p.Account != nil && !unserved {
+ // with no year) would bench the account until someone noticed.
+ // Not a refusal (401, 403) that names no quota: one that did is
+ // failQuota already, and a plain 403 is about the key or the
+ // request, so a cached reading of 98% benched a working account
+ // until its window reset (plugins#243, zygs1083-dotcom)
+ if t := c.full(now); !t.IsZero() && c.p.Account != nil && !unserved && status != http.StatusUnauthorized && status != http.StatusForbidden {
d, r.By, modelBackoff = min(t.Sub(now), longestQuota), "window", false
}
}