Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

gateway: a strict conversation keeps its member past a reply that broke off

yetone7c2f27ccdf6d
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index 796cbeccc..50828cd61 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -147,7 +147,7 @@ recorded as a prediction failure (`noPrediction`).
- A subscription's account whose own model list lacks the model (`Account.Lists`: an Antigravity account's `fetchAvailableModels`, a plugin account's `models`) isn't planned while another of its accounts lists it (`perKeyBarred`'s `left`, `planned.unlisted`, the Routing page's "unlisted, skipped"). It is the next asked when every account of the subscription that lists the model has been asked and one refused it as a model it doesn't serve (`unservedTry`: `modelRefused`, or Code Assist's 404 NOT_FOUND, `codeAssistModelMissing`), before any later member or fallback (`unlistedNext`, `askUnlisted` moves its row from the route's left to its order after the one that refused). A list is fetched once and can be stale: 01huadalang's Gemini 4 was served by one Antigravity account of several, and the one whose list named it refused it with a 404 the agent got while the other could answer. The one that refused rests for that model alone. Not for another refusal (a 403, a quota), a request pinned to an account, a native compaction sealed to one, or a conversation kept strictly on one; a gateway key held to some accounts narrows the unlisted ones too (`allowedCandidates`). `TestUnlistedAccountAskedAfterModelRefused`, `TestUnlistedAccountNotAskedForOtherRefusals`.
- A member whose model is known to take less than the request isn't asked it (`roomy` in [`fallback.go`](../../internal/gateway/fallback.go), in `serve` after the rules, affinity, the image filter and `pinTo`, before `probeOne`; dgqyushen on Discord: OpenCode Zen's free DeepSeek V4.1 Flash, 200K, first in a group and a 1M subscription of it second). It is a check before sending, not a retry: the request's tokens are estimated as the overflow fall-over does (`estimate`, bytes/4 over text, tool arguments and schemas), and a candidate whose window (`windowOf`: the provider's `Contexts`, the catalog, `ContextOf`) is known and no larger is left out of the candidates and of the trace's order. Asked, such a member only refuses, and a refusal that doesn't say why, for a request short of `tooLongUnsaid`'s 1.2 margin, reached the agent as the vendor's error while the next member had room; it also spent a request against the member's limits. A conversation kept on a member by affinity moves once it outgrows it. Nothing is left out when its window isn't known, when the body doesn't parse, when only one model is among the candidates, or when no candidate holds the request: then the first is asked, and its overflow reaches the agent as one, to compact (`withRoom`, `failOverflow` and `tooLongUnsaid` stay for the window the estimate misses, which counts low on code and high on CJK text). A body shorter than four bytes per token of the least window isn't parsed. `count_tokens` (a count on the group's first member, not a send) and System One's decision models (short decision text) don't go through it. Those left out are the last resort: when the last candidate with room fails out of its allowance or credit (`failQuota`/`failCredit`), nothing before it having failed otherwise (`other`), and no account is pinned, the ones left out that aren't resting are asked after all, their rows added to the trace's order, and the one that failed rests as one before the last does. Its overflow tells the agent to compact, where the 429 only told it to wait for an allowance hours away (Kai on Discord: both Claude accounts spent until the next day, the Claude provider's Fallback a Codex model of 272K under a longer conversation, never asked). Another failure (a busy vendor, a bad key) still reaches the agent as it was, the small one not asked. The Routing page shows a member left out and not asked only by its absence from the route's order. `TestMemberTooSmallIsNotAsked`, `TestMemberTooSmallIsTheLastResort`, `TestClaudeAccountsSpentFallBackToCodex`, `TestConversationOutgrowingItsMemberMoves`, `TestRoomyLeavesWhatItCantJudge`.
- Routing decides who goes first only when nobody has answered the conversation yet. Affinity keeps the answerer while its cache is worth it. Auto mode keeps it across turns while the last reply read from the cache and wrote to it at least `cacheWorth` (1024) tokens between them (the usage's `CacheRead` plus `CacheWrite`, passed to `answered`) and the cache isn't cold (`cacheCold`, 5 minutes). Past 5 minutes it stays while the vendor said it keeps at least `cacheWorth` tokens of the conversation for an hour (`hourCache`, #2302): a reply with Anthropic's 1-hour writes (`cache_creation.ephemeral_1h_input_tokens`, the usage's `CacheWrite1h`, as Claude Code asks of a subscription) keeps what it read plus those for `cacheHour`, a later reply on the same account and model that reads all of it starts the hour again, and one that reads less (moved, compacted) drops it. The stick keeps it on disk too (`cacheHour`, `cacheHourAt`), and a group's rules keep the turn's effort the same hour (`keepsEffort`). Vendors that report no 1-hour write are unchanged. `TestAffinityKeepsAnHourCachePastFiveMinutes`, `TestHourCacheAfter`. What was written counts: a first turn reads nothing and writes the whole prompt (Anthropic's `cache_creation_input_tokens`), and the second turn stays to read it rather than moving as `no-cache` and building it again elsewhere (vilicvane on Discord). The stick keeps both, on disk too (`affinity.json`'s `cacheWrite`, absent in a file from before and read as 0, so such a stick is judged by its read alone), and the trace's `Affinity.CacheWrite` lets the Routing page say what was read and what was written. A vendor that keeps a prompt cached without reporting it (OpenAI's automatic prefix cache, Gemini) says neither of a turn that read none, so the prompt that turn was sent (the usage's `Input`) stands for the write and its second turn stays too; a reply that reports no usage at all counts as no cache, as before. `TestAffinityKeepsACacheJustWritten`, `TestAffinityKeepsACacheTheVendorKeptSilently`, `routing-affinity-cache-write.test.cjs`.
-- Strict (`strict`, #1797, miniers) never moves a conversation to another key or account (`who`). Its stick is kept as session's is, and while it is fresh `affine` keeps only the candidates of that `who` (`strictOnly`): its other models stay, so a group's rules still pick among them, but no other account, key or provider is asked, and a resting or spent one is asked anyway. A failure there reaches the agent and rests the account as a non-lone candidate's would (`strictAlone`), so new conversations go elsewhere. Before planning, `strictKeep` takes the stuck account back from what holds new conversations off it: a usage cap (`pl.capped`, `capHeldOnly`), so the conversation goes on past it. An account held for credits it may not spend is answered 429 naming the account and Strict, and nothing reaches the vendor, rather than moved. An account removed, switched off, barred, or not allowed for the calling key is gone, and the conversation moves as under session. The trace says `strict` even with one candidate. `strict_affinity_test.go`, `stays-strict.test.cjs`.
+- Strict (`strict`, #1797, miniers) never moves a conversation to another key or account (`who`). Its stick is kept as session's is, and while it is fresh `affine` keeps only the candidates of that `who` (`strictOnly`): its other models stay, so a group's rules still pick among them, but no other account, key or provider is asked, and a resting or spent one is asked anyway. A failure there reaches the agent and rests the account as a non-lone candidate's would (`strictAlone`), so new conversations go elsewhere. A reply that breaks off after its 200 (#733) forgets the stick under every other mode; under strict it doesn't, so the agent's retry goes back to the same account, member or key, failing there while it fails and going on there once it answers (imwhatiam123 on Discord: a group of Qoder CN and WorkBuddy AI moved a session to WorkBuddy AI, the retry routed as a new conversation; `TestStrictGroupKeepsItsMemberPastABrokenReply`). Before planning, `strictKeep` takes the stuck account back from what holds new conversations off it: a usage cap (`pl.capped`, `capHeldOnly`), so the conversation goes on past it. An account held for credits it may not spend is answered 429 naming the account and Strict, and nothing reaches the vendor, rather than moved. An account removed, switched off, barred, or not allowed for the calling key is gone, and the conversation moves as under session. The trace says `strict` even with one candidate. `strict_affinity_test.go`, `stays-strict.test.cjs`.
- In turn (`rotate`), a conversation nobody has answered yet goes to the key, account or model the fewest other conversations of the provider's model or the group answered in the last `heldFor` (30 minutes) are on (`leastHeld`, #946). Ties, resting ones and a group with a group in it keep routing's order. With session affinity, two sessions of one group so stay on different models.
- A ChatGPT account that holds credits answers past its windows on them; the vendor spends them. That is the default. Set not to spend them (Usage card's *Use credits*, `magpie quota credits <account> off`), the account is held at 100% of a window till it renews, by magpie's latest reading of its windows (`provider.Allowances`, refreshed about every minute), so the other accounts, groups and fallbacks take the request. A cap set on the account (1–99%) takes precedence. Unlike a cap, the hold is at the vendor's own 100%, so an account allowed to spend its resets spends one there, both when every candidate is held and after the last one is refused. A reset made forgets the account's cached allowance (`renewedNow`), so neither hold outlives the windows it started again. One let spend its credits, and holding some by its last `/wham/usage` (`credits.has_credits` with a balance over 0, or `unlimited`; `QuotaWindow.Credits`, `Limit.Credits`, `Allowance.OnCredits`), is not out when a window fills (豆子 and Zhenzhen on Discord): ChatGPT may answer the turn that uses the window up 429 `usage_limit_reached` before it spends the credits, and the next turn on them. So that 429 is asked again at once, once a request, on the same account, before a reset is looked at, on the routed path (`creditsAgain`, `creditsAsked`, `Try.Credits`) and on Codex's own sign-in's relay (`relayOnCredits`), rather than ended for the agent, which ended a running session while a new one worked. Refused again, it rests `fallbackCooldown` (`Rest.By` `credits`) and is read again, not until its window renews, and its full windows rest it nowhere else (`full` is zero for it, `spendsCredits`). Smart, Pace and Least used put such an account first of the spent tier (`routes.Seat.Credits`, `spentBefore`): after those with allowance left, ahead of those used up with nothing to go on with (`Weighed.Credits`, told on the Routing page). Switching to credits early, at a few percent left, was weighed and not done: the vendor spends credits only once the allowance is used up, so it would only waste what is left. `TestCodexCreditsAskedAgainAtTheEdge`, `TestCodexCreditsNoLongRest`, `TestCodexCreditsFirstOfTheSpent`, `TestCreditsFirstOfTheSpent`, `routing-codex-credits-again.test.cjs`.
- Different failures are classified separately: out of credit, out of quota, rate limit and other errors rest for different lengths. A fix must not merge them into one bucket (see #873).
diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go
index 324e79f22..6380e9e82 100644
--- a/internal/gateway/gateway.go
+++ b/internal/gateway/gateway.go
@@ -3657,8 +3657,13 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
// a compaction a rule sent to a model of its own isn't a turn of
// the conversation: it leaves the stick where it was when it
// answers (below), and breaking off mid-reply forgets nothing
- // either (#776's follow-up)
- if hit == nil || !hit.Compact {
+ // either (#776's follow-up). A conversation kept strictly
+ // forgets nothing: it never moves, so the retry goes back to
+ // the one it is on, its failure the agent's, as any other's
+ // there is (imwhatiam123 on Discord: Stays: Strict moved a
+ // session from Qoder CN to WorkBuddy AI after a reply broke
+ // off, the stick gone and the retry routed as a new one)
+ if (hit == nil || !hit.Compact) && aff.Why != "strict" {
unanswered(stuck, c)
}
if hw.looped != "" {
diff --git a/internal/gateway/strict_affinity_test.go b/internal/gateway/strict_affinity_test.go
index 04ef837e5..8197bc5a4 100644
--- a/internal/gateway/strict_affinity_test.go
+++ b/internal/gateway/strict_affinity_test.go
@@ -1,8 +1,11 @@
package gateway
import (
+ "io"
+ "net/http"
"net/http/httptest"
"strings"
+ "sync"
"testing"
"github.com/yetone/magpie/internal/provider"
@@ -146,3 +149,86 @@ func TestStrictAffinityHeldForCreditsIsRefused(t *testing.T) {
t.Fatalf("a new conversation: %d tried %s", code, tried)
}
}
+
+// A strict group's member whose stream breaks off mid-reply, its 200
+// already sent (imwhatiam123 on Discord: a group of Qoder CN and WorkBuddy
+// AI set to Stays: Strict moved a session to WorkBuddy AI mid-way): the
+// conversation isn't forgotten as it is under the other modes (#733), so
+// the agent's retry goes back to the member it is on, failing there as
+// strict says, and goes on there once it answers again. A new conversation
+// goes to the other member while the broken one rests.
+func TestStrictGroupKeepsItsMemberPastABrokenReply(t *testing.T) {
+ fresh(t)
+ const broke = "Unable to reach the model provider"
+ var mu sync.Mutex
+ var tried []string
+ breaking := false
+ up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ io.Copy(io.Discard, r.Body)
+ who := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")
+ mu.Lock()
+ tried = append(tried, who)
+ br := breaking && who == "first"
+ mu.Unlock()
+ w.Header().Set("Content-Type", "text/event-stream")
+ if br {
+ io.WriteString(w, sse(`data: {"id":"c1","choices":[{"index":0,"delta":{"role":"assistant","content":"Let me look"}}]}`,
+ `data: {"error":{"message":"`+broke+`"}}`))
+ return
+ }
+ io.WriteString(w, sse(`data: {"id":"c2","choices":[{"index":0,"delta":{"role":"assistant","content":"from `+who+`"}}]}`,
+ `data: {"id":"c2","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}`,
+ `data: [DONE]`))
+ }))
+ defer up.Close()
+ for _, p := range []provider.Provider{
+ {ID: "qoder-like", Name: "Qoder-like", Key: "first", Chat: up.URL + "/v1", Models: []string{"m"}},
+ {ID: "workbuddy-like", Name: "WorkBuddy-like", Key: "second", Chat: up.URL + "/v1", Models: []string{"m"}},
+ } {
+ if err := provider.Save(p); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if err := provider.SaveGroup(provider.Group{Name: "G", Members: []string{"qoder-like/m", "workbuddy-like/m"}, Routing: provider.Ordered, Affinity: "strict"}); err != nil {
+ t.Fatal(err)
+ }
+ s := New()
+ post := func(session string) (string, string) {
+ mu.Lock()
+ tried = nil
+ mu.Unlock()
+ rec := httptest.NewRecorder()
+ req := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(`{"model":"group/g","stream":true,"messages":[{"role":"user","content":"hi `+session+`"}]}`))
+ req.Header.Set("x-session-id", session)
+ s.Handler().ServeHTTP(rec, req)
+ mu.Lock()
+ defer mu.Unlock()
+ return rec.Body.String(), strings.Join(tried, ",")
+ }
+ if out, got := post("s1"); !strings.Contains(out, "from first") || got != "first" {
+ t.Fatalf("first turn: %s tried %s", out, got)
+ }
+ mu.Lock()
+ breaking = true
+ mu.Unlock()
+ if out, got := post("s1"); !strings.Contains(out, broke) || got != "first" {
+ t.Fatalf("the broken reply: %s tried %s", out, got)
+ }
+ // the agent's retry stays on the member, resting or not
+ if out, got := post("s1"); !strings.Contains(out, broke) || got != "first" {
+ t.Fatalf("strict, the retry after a broken reply moved: %s tried %s", out, got)
+ }
+ if why := lastWhy(t, s); why != "strict" {
+ t.Fatalf("the retry traced as %q, want strict", why)
+ }
+ // a new conversation goes elsewhere meanwhile
+ if out, got := post("s2"); !strings.Contains(out, "from second") || got != "second" {
+ t.Fatalf("a new conversation: %s tried %s", out, got)
+ }
+ mu.Lock()
+ breaking = false
+ mu.Unlock()
+ if out, got := post("s1"); !strings.Contains(out, "from first") || got != "first" {
+ t.Fatalf("strict, the member back: %s tried %s", out, got)
+ }
+}