Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

gateway: a fallback too small for the request is asked once every account with room is out of its allowance (Kai on Discord)

yetoneb14784c03fd9
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index a58a4d6f1..ccea32f78 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -117,7 +117,7 @@ recorded as a prediction failure (`noPrediction`).
- The order of the steps a request's accounts go through, how much the accounts' own order (*Make first*) counts under each routing, and how often an allowance is read are described for users in [How magpie picks an account](../reference.md#how-magpie-picks-an-account). Smart compares renewals window by window, biggest first, each truncated to the hour (`weighRouted`), so the order decides only when every window falls in the same hour. A window's renewal is its reset, or, when what it has left runs out sooner or it has no reset, that moment (`Limit.RunsOut`, from `QuotaWindow.RunsOutAt`; `Allowance.Renewal`): the WorkBuddy plugin (0.1.14) sends it on its Credits window as the soonest end of a package with credits left (cdredfox on X). Only Smart reads it; Weekly pace, the 90%/98% tiers and rests go by the window's share and reset as before. A row on the Routing page says the share used (the fullest window's, `Allowance.For`) with when that same window renews (`Allowance.Fullest`, the trace's `usedRenews`), not the biggest window's: a Factory account 32% into its week and 16% into its month reads "32% used · renews in 3 days" while Smart still ranks it by its month (#1778). A route from before `usedRenews` says when only for an account of one window. Affinity keeps a conversation on an account at 90–97%: only `SpentShareOf` (98%, 100% In order) stops it (`affine`), while the 90% tier (`lowShare`) orders only conversations nobody has answered.
- A turn's end is told to the app for the "Task done alert" notification (itieniu on X, `turndone.go`). The trace follows each agent session's turn (`turns`, keyed by agent and session): a request is of the turn unless it has a `Kind` or is a Claude Code subagent's (`agentTurnOf`), and its reply is scanned on the way to the agent for a tool call in any wire format (`toolScan`, a JSON value such as `"type":"tool_use"` or `"finish_reason":"tool_calls"`, never a key or text). The turn is done when no request of it is going, the last reply was whole, under 400 and called no tool, and nothing more came for `turnQuiet` (4s); a session idle past `turnStale` (30m) starts over. Only a turn that ran at least `settings.TurnDoneAlert` seconds is told (`TurnDone`, set by the GUI; nil under `magpie serve`, so nothing is followed). The ChatGPT backend path (`codexBackend`) is followed the same way. `TestTurnDoneSaidAtTheTurnsEnd`, `TestTurnDoneCodexBackend`, `TestToolScan`, `turn-done.test.cjs`.
- A resting candidate is tried last (`restLast`). Of the resting candidates, a request asks only the one whose rest ends soonest, the first in order on a tie (`probeOne`, #1799). The others aren't asked by that request, so a member that is down isn't asked by every request its rest was meant to spare it from, and a group that is down whole takes one upstream request per request, not one per member. Every ready candidate is still asked in its order. A lone candidate is tried even while it rests, because there is no other. The trace still lists every member, the resting ones with their rest. Only rests are read, and a failure about the request's content or its model rests no member (a model refusal rests only `restID`), so neither leaves a member out. The same applies in `count_tokens`, embeddings and rerank, image accounts and a group's System One decision models. A Codex reset (`autoReset`) still looks over every candidate. This is like AxonHub's error-aware load balancing: the rest and its backoff are the penalty that decays, and the member back soonest is the one asked to show it is back. Keys out of their allowance aren't asked even so, not even the one back soonest or a lone one (#1772: four Ollama Cloud keys out for the week, each asked again and refused on every request). The refusal given again says when they are asked again on this magpie's clock with its offset (`heldClock`: "20:12 UTC+8", "Oct 16 09:00 UTC+8"), as do the gateway's other held errors (a capped window, a spent searcher or describer): its reader may be a Remote magpie's user on another clock. When every candidate left is a key (not a subscription account, whose resets `autoReset` may spend) resting out of quota or credit after a 429 or 402, none is asked. The first one's refusal is given again (`quotaHeld`, its words kept in the rest's `said`), saying when it is asked again. The last key left rests too when it is refused so, as the keys before it do. Any 429 that reaches the agent without a vendor `Retry-After`, while every candidate tried is resting, carries one: the seconds until the first of them is back (`soonestBack`). A quota refusal with no word of when it is back rests `quotaRest` (15 minutes), or the provider's own `quotaRest` in minutes (`Provider.QuotaRestFor`, at most `MaxQuotaRest`, 8 days; `magpie provider quota-rest <id> 7d`), or until the time of day the user said its allowance comes back (`QuotaResetAt`, `Provider.OwnQuotaRest`; `magpie provider quota-rest <id> until 08:00`, or `until 00:00+08:00` on the vendor's clock; #2008, a free tier's "today's free-model token quota"). Whichever of the two was set last counts. A time the vendor names stands over it, and so does its `Retry-After`, unless that was more than `longestWait` (an hour) and so cut to it while the user's own rest is longer: then the user's stands (`TestQuotaRestsUntilTheTimeOfDaySet`). A key listed in `keys` as well as in `key` is one candidate (`KeysOn`), not asked twice. `TestAllRestingAnswersFromRest`, `TestOwnQuotaRest`, `TestQuotaHeldOnlyForKeysOutOfAllowance`, `TestFirstKeyListedTwiceAskedOnce`.
-- A member whose model is known to take less than the request isn't asked it (`roomy` in [`fallback.go`](../../internal/gateway/fallback.go), in `serve` after the rules, affinity, the image filter and `pinTo`, before `probeOne`; dgqyushen on Discord: OpenCode Zen's free DeepSeek V4.1 Flash, 200K, first in a group and a 1M subscription of it second). It is a check before sending, not a retry: the request's tokens are estimated as the overflow fall-over does (`estimate`, bytes/4 over text, tool arguments and schemas), and a candidate whose window (`windowOf`: the provider's `Contexts`, the catalog, `ContextOf`) is known and no larger is left out of the candidates and of the trace's order. Asked, such a member only refuses, and a refusal that doesn't say why, for a request short of `tooLongUnsaid`'s 1.2 margin, reached the agent as the vendor's error while the next member had room; it also spent a request against the member's limits. A conversation kept on a member by affinity moves once it outgrows it. Nothing is left out when its window isn't known, when the body doesn't parse, when only one model is among the candidates, or when no candidate holds the request: then the first is asked, and its overflow reaches the agent as one, to compact (`withRoom`, `failOverflow` and `tooLongUnsaid` stay for the window the estimate misses, which counts low on code and high on CJK text). A body shorter than four bytes per token of the least window isn't parsed. `count_tokens` (a count on the group's first member, not a send) and System One's decision models (short decision text) don't go through it. The Routing page shows a member left out so only by its absence from the route's order. `TestMemberTooSmallIsNotAsked`, `TestConversationOutgrowingItsMemberMoves`, `TestRoomyLeavesWhatItCantJudge`.
+- A member whose model is known to take less than the request isn't asked it (`roomy` in [`fallback.go`](../../internal/gateway/fallback.go), in `serve` after the rules, affinity, the image filter and `pinTo`, before `probeOne`; dgqyushen on Discord: OpenCode Zen's free DeepSeek V4.1 Flash, 200K, first in a group and a 1M subscription of it second). It is a check before sending, not a retry: the request's tokens are estimated as the overflow fall-over does (`estimate`, bytes/4 over text, tool arguments and schemas), and a candidate whose window (`windowOf`: the provider's `Contexts`, the catalog, `ContextOf`) is known and no larger is left out of the candidates and of the trace's order. Asked, such a member only refuses, and a refusal that doesn't say why, for a request short of `tooLongUnsaid`'s 1.2 margin, reached the agent as the vendor's error while the next member had room; it also spent a request against the member's limits. A conversation kept on a member by affinity moves once it outgrows it. Nothing is left out when its window isn't known, when the body doesn't parse, when only one model is among the candidates, or when no candidate holds the request: then the first is asked, and its overflow reaches the agent as one, to compact (`withRoom`, `failOverflow` and `tooLongUnsaid` stay for the window the estimate misses, which counts low on code and high on CJK text). A body shorter than four bytes per token of the least window isn't parsed. `count_tokens` (a count on the group's first member, not a send) and System One's decision models (short decision text) don't go through it. Those left out are the last resort: when the last candidate with room fails out of its allowance or credit (`failQuota`/`failCredit`), nothing before it having failed otherwise (`other`), and no account is pinned, the ones left out that aren't resting are asked after all, their rows added to the trace's order, and the one that failed rests as one before the last does. Its overflow tells the agent to compact, where the 429 only told it to wait for an allowance hours away (Kai on Discord: both Claude accounts spent until the next day, the Claude provider's Fallback a Codex model of 272K under a longer conversation, never asked). Another failure (a busy vendor, a bad key) still reaches the agent as it was, the small one not asked. The Routing page shows a member left out and not asked only by its absence from the route's order. `TestMemberTooSmallIsNotAsked`, `TestMemberTooSmallIsTheLastResort`, `TestClaudeAccountsSpentFallBackToCodex`, `TestConversationOutgrowingItsMemberMoves`, `TestRoomyLeavesWhatItCantJudge`.
- Routing decides who goes first only when nobody has answered the conversation yet. Affinity keeps the answerer while its cache is worth it. Auto mode keeps it across turns while the last reply read from the cache and wrote to it at least `cacheWorth` (1024) tokens between them (the usage's `CacheRead` plus `CacheWrite`, passed to `answered`) and the cache isn't cold (`cacheCold`, 5 minutes). Past 5 minutes it stays while the vendor said it keeps at least `cacheWorth` tokens of the conversation for an hour (`hourCache`, #2302): a reply with Anthropic's 1-hour writes (`cache_creation.ephemeral_1h_input_tokens`, the usage's `CacheWrite1h`, as Claude Code asks of a subscription) keeps what it read plus those for `cacheHour`, a later reply on the same account and model that reads all of it starts the hour again, and one that reads less (moved, compacted) drops it. The stick keeps it on disk too (`cacheHour`, `cacheHourAt`), and a group's rules keep the turn's effort the same hour (`keepsEffort`). Vendors that report no 1-hour write are unchanged. `TestAffinityKeepsAnHourCachePastFiveMinutes`, `TestHourCacheAfter`. What was written counts: a first turn reads nothing and writes the whole prompt (Anthropic's `cache_creation_input_tokens`), and the second turn stays to read it rather than moving as `no-cache` and building it again elsewhere (vilicvane on Discord). The stick keeps both, on disk too (`affinity.json`'s `cacheWrite`, absent in a file from before and read as 0, so such a stick is judged by its read alone), and the trace's `Affinity.CacheWrite` lets the Routing page say what was read and what was written. A vendor that keeps a prompt cached without reporting it (OpenAI's automatic prefix cache, Gemini) says neither of a turn that read none, so the prompt that turn was sent (the usage's `Input`) stands for the write and its second turn stays too; a reply that reports no usage at all counts as no cache, as before. `TestAffinityKeepsACacheJustWritten`, `TestAffinityKeepsACacheTheVendorKeptSilently`, `routing-affinity-cache-write.test.cjs`.
- Strict (`strict`, #1797, miniers) never moves a conversation to another key or account (`who`). Its stick is kept as session's is, and while it is fresh `affine` keeps only the candidates of that `who` (`strictOnly`): its other models stay, so a group's rules still pick among them, but no other account, key or provider is asked, and a resting or spent one is asked anyway. A failure there reaches the agent and rests the account as a non-lone candidate's would (`strictAlone`), so new conversations go elsewhere. Before planning, `strictKeep` takes the stuck account back from what holds new conversations off it: a usage cap (`pl.capped`, `capHeldOnly`), so the conversation goes on past it. An account held for credits it may not spend is answered 429 naming the account and Strict, and nothing reaches the vendor, rather than moved. An account removed, switched off, barred, or not allowed for the calling key is gone, and the conversation moves as under session. The trace says `strict` even with one candidate. `strict_affinity_test.go`, `stays-strict.test.cjs`.
- In turn (`rotate`), a conversation nobody has answered yet goes to the key, account or model the fewest other conversations of the provider's model or the group answered in the last `heldFor` (30 minutes) are on (`leastHeld`, #946). Ties, resting ones and a group with a group in it keep routing's order. With session affinity, two sessions of one group so stay on different models.
@@ -713,7 +713,7 @@ the row of the entry that id reaches, with its label, under the bare id.
```sh
go test -tags nogui ./internal/gateway -run 'TestGroupAsksOneRestingMemberNotEach|TestLoneRestingMemberStillAsked|TestRetrievalGroupAsksOneRestingMember|TestProbeOne|TestLoop|TestLongReplyThatDoesNotLoop|TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestModelNotInPlanRestsTheModelNotTheKey|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestWindowCapsHoldTheAccount|TestAccountCap|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestSearchFollowsTheSearchersProxy|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestProtection|TestSearchedConversation|TestUnsearchedConversationUnchanged|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestClaudeAccountBackOnceItsWindowRenews|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestCodexHeardWithinItsIdleTimeout|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused|TestAIStudioThinking|TestGeminiThoughtsThroughLocalProxy|TestGeminiRefusedLevel|TestGeminiThinkingOffAtItsLowest|TestThinkingOff|TestTranslatedEffortLoggedAsSent|TestTranslatedThinkingOffRefused|TestSub2APIThinkingOffRefused|TestRefusesThinkingOff|TestClaudeFiveFiveThinkingOffOnTheWire|TestSamplingAsTheModelTakesIt|TestHaikuFiveFiveSamplingOnTheWire|TestGeminiThoughtsMarkedInExtraContent|TestGeminiThinkingLevelRefused|TestGeminiMinimalRefusedTwiceAtOnce|TestPassthroughGeminiMinimalRefused|TestPassthroughEffortNoneRefused|TestPassthroughMinimalRefusedNoneTaken|TestOtherOffNamed|TestClaudeSignInNotToldItsSignInFailed|TestSignInWriterReleasesA403|TestSignInBetaStaysWithMagpie|TestStructuredOutput|TestResponsesTextKeepsVerbosityBesideFormat|TestRefusedFormatToldInWords|RPM|TestQueued|TestCustomGeminiUpstream|TestWholeNums|TestGrokCalls|TestOnlyGrokCalls|TestGrokModel|TestGrokPluginCalls'
-go test -tags nogui ./internal/gateway -run 'TestMemberTooSmallIsNotAsked|TestConversationOutgrowingItsMemberMoves|TestRoomyLeavesWhatItCantJudge|TestPayloadTooLarge|TestOverflow|TestContextOverflow|TestRelayInputCap|TestEveryVendorsOverflow'
+go test -tags nogui ./internal/gateway -run 'TestMemberTooSmallIsNotAsked|TestMemberTooSmallIsTheLastResort|TestClaudeAccountsSpentFallBackToCodex|TestConversationOutgrowingItsMemberMoves|TestRoomyLeavesWhatItCantJudge|TestPayloadTooLarge|TestOverflow|TestContextOverflow|TestRelayInputCap|TestEveryVendorsOverflow'
go test -tags nogui ./internal/gateway -run 'TestVertex'
go test -tags nogui ./internal/gateway -run 'TestBedrock'
go test -tags nogui ./internal/gateway -run 'TestPublicAddr|TestEditImageURL|TestPrivateDialer|TestFetchPrivateEntries|TestVendorImageURLAtPrivateAddress|TestVideoContentAtPrivateAddress'
diff --git a/internal/gateway/claude_codex_fallback_test.go b/internal/gateway/claude_codex_fallback_test.go
new file mode 100644
index 000000000..75afcf215
--- /dev/null
+++ b/internal/gateway/claude_codex_fallback_test.go
@@ -0,0 +1,159 @@
+package gateway
+
+import (
+ "encoding/base64"
+ "encoding/json"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "runtime"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/yetone/magpie/internal/provider"
+ "github.com/yetone/magpie/internal/testenv"
+)
+
+// Kai on Discord (0.1.1278): Claude Code asks claude/claude-opus-5-5 at
+// high effort, both Claude accounts are out of their limit, and the Claude
+// provider's own Fallback, a Codex subscription model, takes the request —
+// streamed or not.
+func TestClaudeAccountsSpentFallBackToCodex(t *testing.T) {
+ if runtime.GOOS == "windows" {
+ t.Skip("a shell script stands in for Claude Code")
+ }
+ fresh(t)
+ home := t.TempDir()
+ t.Setenv("HOME", home)
+ t.Setenv("USERPROFILE", home)
+ t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, ".config"))
+ t.Setenv("XDG_CACHE_HOME", filepath.Join(home, ".cache"))
+ far := time.Now().Add(24 * time.Hour).UnixMilli()
+ oauth := func(tok, plan string) map[string]any {
+ return map[string]any{"claudeAiOauth": map[string]any{"accessToken": tok, "refreshToken": "r-" + tok, "expiresAt": far, "subscriptionType": plan}}
+ }
+ os.MkdirAll(filepath.Join(home, ".claude"), 0o755)
+ os.WriteFile(filepath.Join(home, ".claude", ".credentials.json"), mustJSON(oauth("tok-primary", "max")), 0o600)
+ os.WriteFile(filepath.Join(home, ".claude.json"), mustJSON(map[string]any{
+ "oauthAccount": map[string]any{"emailAddress": "[email protected]", "accountUuid": "u-me"}}), 0o600)
+ claims := func(m map[string]any) string {
+ b, _ := json.Marshal(m)
+ return "h." + base64.RawURLEncoding.EncodeToString(b) + ".s"
+ }
+ os.MkdirAll(filepath.Join(home, ".codex"), 0o755)
+ os.WriteFile(filepath.Join(home, ".codex", "auth.json"), mustJSON(map[string]any{"auth_mode": "chatgpt", "tokens": map[string]any{
+ "id_token": claims(map[string]any{"email": "[email protected]"}),
+ "access_token": claims(map[string]any{"exp": time.Now().Add(time.Hour).Unix(), "who": "acct-1"}),
+ "refresh_token": "r-acct-1", "account_id": "acct-1"}}), 0o600)
+ os.MkdirAll(filepath.Dir(provider.Path()), 0o755)
+ os.WriteFile(filepath.Join(filepath.Dir(provider.Path()), "logins.json"), mustJSON([]map[string]any{
+ {"agent": "claude", "user": "[email protected]", "plan": "max", "on": true, "seen": time.Now(), "auth": oauth("tok-me", "max")},
+ {"agent": "claude", "user": "[email protected]", "plan": "pro", "on": true, "seen": time.Now(), "auth": oauth("tok-pro", "pro")},
+ }), 0o600)
+ dir := t.TempDir()
+ log := filepath.Join(dir, "log")
+ // every account is out of its limit: the CLI says so in a result
+ script := `#!/bin/sh
+if [ "$1" = auth ]; then echo '{"loggedIn":true,"authMethod":"oauth_token","apiProvider":"firstParty"}'; exit 0; fi
+TOK=; [ -n "$CLAUDE_CONFIG_DIR" ] && TOK=$(sed -n 's/.*"accessToken": *"\([^"]*\)".*/\1/p' "$CLAUDE_CONFIG_DIR/.credentials.json")
+echo "${TOK:-own}" >> ` + log + `
+while read -r line; do
+ echo '{"type":"rate_limit_event","rate_limit_info":{"status":"rejected","rateLimitType":"five_hour","resetsAt":1790700000}}'
+ echo '{"type":"assistant","message":{"id":"x","model":"<synthetic>","role":"assistant","content":[{"type":"text","text":"You'"'"'ve hit your limit · resets 5:58am"}]},"error":"rate_limit"}'
+ echo '{"type":"result","subtype":"success","is_error":true,"result":"You'"'"'ve hit your limit · resets 5:58am"}'
+done
+`
+ testenv.Program(t, filepath.Join(dir, "claude"), script)
+ t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
+
+ var mu sync.Mutex
+ var codex []string
+ up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ b, _ := io.ReadAll(r.Body)
+ mu.Lock()
+ codex = append(codex, string(b))
+ mu.Unlock()
+ w.Header().Set("Content-Type", "text/event-stream")
+ io.WriteString(w, sse(
+ `data: {"type":"response.created","response":{"id":"r1","model":"gpt-6.1-sol"}}`,
+ `data: {"type":"response.output_item.added","output_index":0,"item":{"type":"message","id":"m1","role":"assistant","content":[]}}`,
+ `data: {"type":"response.output_text.delta","item_id":"m1","output_index":0,"content_index":0,"delta":"pong"}`,
+ `data: {"type":"response.output_item.done","output_index":0,"item":{"type":"message","id":"m1","role":"assistant","content":[{"type":"output_text","text":"pong"}]}}`,
+ `data: {"type":"response.completed","response":{"id":"r1","status":"completed","output":[{"type":"message","id":"m1","role":"assistant","content":[{"type":"output_text","text":"pong"}]}],"usage":{"input_tokens":7,"output_tokens":1}}}`))
+ }))
+ defer up.Close()
+ old := provider.CodexBase
+ provider.CodexBase = up.URL + "/backend-api/codex"
+ defer func() { provider.CodexBase = old }()
+ provider.ForgetAccounts()
+ t.Cleanup(provider.ForgetAccounts)
+ if err := provider.Save(provider.Provider{ID: "claude", Fallback: []string{"codex/gpt-6.1-sol"}}); err != nil {
+ t.Fatal(err)
+ }
+ p, err := provider.Find("claude")
+ if err != nil || len(p.Fallback) != 1 {
+ t.Fatalf("claude's fallback not kept: %+v %v", p.Fallback, err)
+ }
+
+ // Codex holds a ChatGPT plan's conversation to 272K; Opus holds 1M
+ if err := provider.Save(provider.Provider{ID: "codex", Contexts: map[string]int{"gpt-6.1-sol": 272000}}); err != nil {
+ t.Fatal(err)
+ }
+ long := strings.Repeat("word ", 300000) // ~375K tokens by estimate
+ for _, tc := range []struct {
+ stream, text string
+ again bool
+ }{
+ {"true", "ping", false}, {"false", "ping", false}, {"true", "ping", true},
+ // a conversation past Codex's window: the fallback is still asked
+ // when every account with room is out of its limit, for the agent
+ // to be told to compact rather than wait hours (Kai)
+ {"true", long, false}, {"false", long, false},
+ } {
+ stream := tc.stream
+ restingUntil.Lock()
+ restingUntil.m = map[string]time.Time{}
+ restingUntil.Unlock()
+ os.Remove(log)
+ mu.Lock()
+ codex = nil
+ mu.Unlock()
+ s := New()
+ if tc.again {
+ for k := 0; k < 1; k++ {
+ rq := httptest.NewRequest("POST", "/v1/messages", strings.NewReader(`{"model":"claude/claude-opus-5-5","max_tokens":50,"stream":true,"messages":[{"role":"user","content":"ping"}]}`))
+ rq.Header.Set("User-Agent", "claude-cli/2.1.0 (external, cli)")
+ s.Handler().ServeHTTP(httptest.NewRecorder(), rq)
+ }
+ mu.Lock()
+ codex = nil
+ mu.Unlock()
+ }
+ done := make(chan struct{})
+ rec := httptest.NewRecorder()
+ go func() {
+ defer close(done)
+ req := httptest.NewRequest("POST", "/v1/messages", strings.NewReader(`{"model":"claude/claude-opus-5-5","max_tokens":50,"stream":`+stream+`,"thinking":{"type":"adaptive"},"output_config":{"effort":"high"},"messages":[{"role":"user","content":"`+tc.text+`"}]}`))
+ req.Header.Set("User-Agent", "claude-cli/2.1.0 (external, cli)")
+ s.Handler().ServeHTTP(rec, req)
+ }()
+ select {
+ case <-done:
+ case <-time.After(30 * time.Second):
+ t.Fatalf("stream=%s: no answer", stream)
+ }
+ b, _ := os.ReadFile(log)
+ mu.Lock()
+ got := len(codex)
+ mu.Unlock()
+ if rec.Code != 200 || !strings.Contains(rec.Body.String(), "pong") || got == 0 {
+ j, _ := json.Marshal(s.Trace(t.Context(), 0, 0))
+ route := string(j)
+ t.Errorf("stream=%s %d bytes again=%v: status %d: %.400s (claude runs %q, codex asked %d)\nroute %.3000s", stream, len(tc.text), tc.again, rec.Code, rec.Body.String(), b, got, route)
+ }
+ }
+}
diff --git a/internal/gateway/fallback.go b/internal/gateway/fallback.go
index 47292b731..c34871b28 100644
--- a/internal/gateway/fallback.go
+++ b/internal/gateway/fallback.go
@@ -1405,9 +1405,18 @@ func holds(c candidate, tokens int) bool {
// request that doesn't parse leaves every one, and so does one no
// candidate holds: the first is asked, and the agent told it overflowed,
// to compact.
-func roomy(cs []candidate, pl planned, from provider.Protocol, body []byte) ([]candidate, planned) {
+//
+// Those left out are handed back (small, their rows in smallW when the
+// order has one each), kept for the last resort: when every candidate
+// with room is out of its allowance, a fallback too small is asked all
+// the same (Kai on Discord: both Claude accounts at their limit until
+// the next day, the Claude provider's Fallback a Codex model of 272K
+// under a longer conversation, left out without a word, and Claude Code
+// handed the 429). Its overflow tells the agent to compact, after which
+// the fallback holds it; the 429 told it only to wait for hours.
+func roomy(cs []candidate, pl planned, from provider.Protocol, body []byte) (out []candidate, _ planned, small []candidate, smallW []Weighed) {
if len(cs) < 2 {
- return cs, pl
+ return cs, pl, nil, nil
}
served := sync.OnceValue(provider.Served) // the catalog, read once
least, windows := 0, map[string]int{}
@@ -1425,18 +1434,21 @@ func roomy(cs []candidate, pl planned, from provider.Protocol, body []byte) ([]c
// one model has nothing to choose between, and a body shorter than
// the least window can't estimate past it: neither is parsed
if len(windows) < 2 || least == 0 || len(body)/4 < least {
- return cs, pl
+ return cs, pl, nil, nil
}
req, err := parse(from, body)
if err != nil {
- return cs, pl
+ return cs, pl, nil, nil
}
tokens := estimate(req)
- var out []candidate
var order []Weighed
parallel := len(pl.order) == len(cs)
for i, c := range cs {
if w := windows[key(c)]; w > 0 && tokens >= w {
+ small = append(small, c)
+ if parallel {
+ smallW = append(smallW, pl.order[i])
+ }
continue
}
out = append(out, c)
@@ -1445,12 +1457,12 @@ func roomy(cs []candidate, pl planned, from provider.Protocol, body []byte) ([]c
}
}
if len(out) == 0 || len(out) == len(cs) {
- return cs, pl
+ return cs, pl, nil, nil
}
if parallel {
pl.order = order
}
- return out, pl
+ return out, pl, small, smallW
}
// matesFirst puts first, of the candidates left, the other keys or
diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go
index 3894c35f3..63fe7f7e3 100644
--- a/internal/gateway/gateway.go
+++ b/internal/gateway/gateway.go
@@ -2192,8 +2192,9 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
}
}
// a member whose model is known to take less than the request isn't
- // asked: it would only refuse it
- cands, pl = roomy(cands, pl, from, body)
+ // asked: it would only refuse it. Those left out are the last resort,
+ // once every one with room is out of its allowance (small, below)
+ cands, pl, small, smallW := roomy(cands, pl, from, body)
// of those resting, only the one back soonest is asked (#1799); the
// whole list still finds a Codex account to spend a reset on
all := cands
@@ -3230,6 +3231,38 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
continue
}
}
+ if f := failure(hw.code(), hw.errBody()); last && len(small) > 0 && pin == "" && other == nil && hw.failed() && !hw.passing && !hw.turnedAway && !hw.refused && !outgrew &&
+ (f == failQuota || f == failCredit || try.Fail == failQuota || try.Fail == failCredit) {
+ // every one with room is out of its allowance: those roomy
+ // left out as too small for the request are asked after all,
+ // those not resting, for the agent to be told to compact
+ // rather than to wait for an allowance hours away (Kai on
+ // Discord: Claude's two accounts spent, its Fallback a Codex
+ // model of 272K). This one rests, as one before the last does
+ var next []candidate
+ var nextW []Weighed
+ for k, x := range small {
+ if _, resting := restingOf(x); resting {
+ continue
+ }
+ next = append(next, x)
+ if len(smallW) == len(small) {
+ nextW = append(nextW, smallW[k])
+ }
+ }
+ small = nil
+ if len(next) > 0 {
+ rest := s.restAfterMarked(c, hw.code(), hw.header, hw.errBody(), hw.sharedPool)
+ try.Fail, try.Rest = rest.Why, &rest
+ s.trace.update(tr, func(t *Route) {
+ t.Tries[len(t.Tries)-1] = try
+ t.Order = append(slices.Clip(t.Order), nextW...)
+ })
+ skipped = append(skipped, c.label()+": "+call.Error)
+ cands = append(cands[:len(cands):len(cands)], next...)
+ continue
+ }
+ }
if last && !held && !hw.passing && hw.failed() && c.p.Account == nil && quotaStatus(hw.code()) && (try.Fail == failQuota || try.Fail == failCredit || failure(hw.code(), hw.errBody()) == failQuota || failure(hw.code(), hw.errBody()) == failCredit) {
// the last key left is out of its allowance: it rests as the
// ones before it did, for the next request to be answered
diff --git a/internal/gateway/roomy_test.go b/internal/gateway/roomy_test.go
index 1673eb553..6c0108d77 100644
--- a/internal/gateway/roomy_test.go
+++ b/internal/gateway/roomy_test.go
@@ -66,6 +66,43 @@ func TestMemberTooSmallIsNotAsked(t *testing.T) {
}
}
+// A member left out as too small is the last resort: asked after all
+// once every member with room is out of its allowance, for the agent to
+// be told to compact (Kai on Discord: both Claude accounts spent until
+// the next day, the Claude provider's Fallback a Codex model of 272K),
+// but not for a failure of another kind, whose error the agent is told
+// as before.
+func TestMemberTooSmallIsTheLastResort(t *testing.T) {
+ fresh(t)
+ free := &scripted{replies: []reply{{200, "", chatOK}}}
+ paid := &scripted{replies: []reply{{401, "", `{"error":{"message":"Invalid API key"}}`}}}
+ roomyMember(t, "free", 2000, free)
+ roomyMember(t, "paid", 10000, paid)
+ if err := provider.SaveGroup(provider.Group{Name: "G", Members: []string{"paid/m", "free/m"}, Routing: provider.Ordered}); err != nil {
+ t.Fatal(err)
+ }
+ s := New()
+ past := strings.Repeat("word ", 1760) // about 2200 tokens as estimated
+ code, body := postAs(t, s, "", `{"model":"group/g","messages":[{"role":"user","content":"`+past+`"}]}`)
+ if code != 401 || free.n != 0 || paid.n != 1 {
+ t.Fatalf("paid's key refused: %d %s (free %d, paid %d)", code, body, free.n, paid.n)
+ }
+
+ paid.replies = []reply{{429, "", `{"error":{"code":"insufficient_quota","message":"You exceeded your current quota, please check your plan and billing details."}}`}}
+ paid.n = 0
+ code, body = postAs(t, s, "", `{"model":"group/g","messages":[{"role":"user","content":"`+past+`"}]}`)
+ r := lastRoute(s)
+ if code != 200 || free.n != 1 || paid.n != 1 {
+ t.Fatalf("paid out of its allowance: %d %s (free %d, paid %d)", code, body, free.n, paid.n)
+ }
+ if len(r.Tries) != 2 || r.Tries[0].ID != "paid" || r.Tries[0].Rest == nil || r.Tries[1].ID != "free" {
+ t.Fatalf("tries %+v", r.Tries)
+ }
+ if !slices.ContainsFunc(r.Order, func(w Weighed) bool { return w.Provider == "free" }) {
+ t.Fatalf("the free member asked isn't in the order: %+v", r.Order)
+ }
+}
+
// A conversation that began on the free member, and is kept there by
// affinity, moves once it has grown past that member's window.
func TestConversationOutgrowingItsMemberMoves(t *testing.T) {
@@ -125,13 +162,17 @@ func TestRoomyLeavesWhatItCantJudge(t *testing.T) {
{p: provider.Provider{ID: "big", Contexts: map[string]int{"m": 10000}}, model: "m", rest: "big"},
}
pl := planned{order: []Weighed{{ID: "small"}, {ID: "big"}}}
- got, gotPl := roomy(cs, pl, provider.Chat, []byte(`{"messages":[`+strings.Repeat(`"word `, 4000)))
- if len(got) != 2 || len(gotPl.order) != 2 {
+ got, gotPl, out, outW := roomy(cs, pl, provider.Chat, []byte(`{"messages":[`+strings.Repeat(`"word `, 4000)))
+ if len(got) != 2 || len(gotPl.order) != 2 || out != nil || outW != nil {
t.Fatalf("unparsed: %d candidates, order %+v", len(got), gotPl.order)
}
- // and one that does leaves the small one out, its order with it
- got, gotPl = roomy(cs, pl, provider.Chat, []byte(`{"model":"g","messages":[{"role":"user","content":"`+big+`"}]}`))
+ // and one that does leaves the small one out, its order with it,
+ // both handed back for the last resort
+ got, gotPl, out, outW = roomy(cs, pl, provider.Chat, []byte(`{"model":"g","messages":[{"role":"user","content":"`+big+`"}]}`))
if len(got) != 1 || got[0].rest != "big" || len(gotPl.order) != 1 || gotPl.order[0].ID != "big" {
t.Fatalf("parsed: %+v, order %+v", got, gotPl.order)
}
+ if len(out) != 1 || out[0].rest != "small" || len(outW) != 1 || outW[0].ID != "small" {
+ t.Fatalf("left out: %+v, rows %+v", out, outW)
+ }
}