Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

gateway: a conversation on a ChatGPT account stays on it for an hour unread, not five minutes (leelantao)

yetoneb5582912af3b
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index 9bbff9044..5efff2be1 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -198,7 +198,7 @@ recorded as a prediction failure (`noPrediction`).
- A provider whose keys are on different plans (`KeyAccount.Plan`, `PlansMixed`; yckonnnn on X) sends each key to its plan's endpoints: every key path takes its copy through `WithKey`, which moves it there (`atKeyPlan`), on the first try and on a failover alike. `perKeyBarred` leaves out a key whose plan's list lacks the model (`PlanServing`: a pay-as-you-go model id at the Agent Plan's endpoint only answers 404), not shown in the trace; only when no key's plan has it are those keys asked. Embeddings and drawing go through the same `perKeyBarred` and `WithKey`; a video started on a mixed provider names its key in the task id (`videoStart.onKey`, `keyMark`) so its poll goes to that key's plan. `TestKeyPlansEachToItsOwnEndpoint`.
- A subscription's account whose own model list lacks the model (`Account.Lists`: an Antigravity account's `fetchAvailableModels`, a plugin account's `models`) isn't planned while another of its accounts lists it (`perKeyBarred`'s `left`, `planned.unlisted`, the Routing page's "unlisted, skipped"). It is the next asked when every account of the subscription that lists the model has been asked and one refused it as a model it doesn't serve (`unservedTry`: `modelRefused`, or Code Assist's 404 NOT_FOUND, `codeAssistModelMissing`), before any later member or fallback (`unlistedNext`, `askUnlisted` moves its row from the route's left to its order after the one that refused). A list is fetched once and can be stale: 01huadalang's Gemini 4 was served by one Antigravity account of several, and the one whose list named it refused it with a 404 the agent got while the other could answer. The one that refused rests for that model alone. Not for another refusal (a 403, a quota), a request pinned to an account, a native compaction sealed to one, or a conversation kept strictly on one; a gateway key held to some accounts narrows the unlisted ones too (`allowedCandidates`). `TestUnlistedAccountAskedAfterModelRefused`, `TestUnlistedAccountNotAskedForOtherRefusals`.
- A member whose model is known to take less than the request isn't asked it (`roomy` in [`fallback.go`](../../internal/gateway/fallback.go), in `serve` after the rules, affinity, the image filter and `pinTo`, before `probeOne`; dgqyushen on Discord: OpenCode Zen's free DeepSeek V4.1 Flash, 200K, first in a group and a 1M subscription of it second). It is a check before sending, not a retry: the request's tokens are estimated as the overflow fall-over does (`estimate`, bytes/4 over text, tool arguments and schemas), and a candidate whose window (`windowOf`: the provider's `Contexts`, the catalog, `ContextOf`) is known and no larger is left out of the candidates and of the trace's order, and listed in the trace's `left` with `window` set to that window. Asked, such a member only refuses, and a refusal that doesn't say why, for a request short of `tooLongUnsaid`'s 1.2 margin, reached the agent as the vendor's error while the next member had room; it also spent a request against the member's limits. A conversation kept on a member by affinity moves once it outgrows it. Nothing is left out when its window isn't known, when the body doesn't parse, when only one model is among the candidates, or when no candidate holds the request: then the first is asked, and its overflow reaches the agent as one, to compact (`withRoom`, `failOverflow` and `tooLongUnsaid` stay for the window the estimate misses, which counts low on code and high on CJK text). A body shorter than four bytes per token of the least window isn't parsed. `count_tokens` (a count on the group's first member, not a send) and System One's decision models (short decision text) don't go through it. Those left out are the last resort: when the last candidate with room fails out of its allowance or credit (`failQuota`/`failCredit`), nothing before it having failed otherwise (`other`), and no account is pinned, the ones left out that aren't resting are asked after all, their rows moved from the trace's `left` to its order, and the one that failed rests as one before the last does. Its overflow tells the agent to compact, where the 429 only told it to wait for an allowance hours away (Kai on Discord: both Claude accounts spent until the next day, the Claude provider's Fallback a Codex model of 272K under a longer conversation, never asked). Another failure (a busy vendor, a bad key) still reaches the agent as it was, the small one not asked. The Routing page keeps a row for a member left out and not asked, saying the request is longer than its window and that it was passed over, and its why list says so (#3749; it used to show it only by its absence from the order). `TestMemberTooSmallIsNotAsked`, `TestMemberTooSmallIsTheLastResort`, `TestClaudeAccountsSpentFallBackToCodex`, `TestConversationOutgrowingItsMemberMoves`, `TestRoomyLeavesWhatItCantJudge`.
-- Routing decides who goes first only when nobody has answered the conversation yet. Affinity keeps the answerer while its cache is worth it. Auto mode keeps it across turns while the last reply read from the cache and wrote to it at least `cacheWorth` (1024) tokens between them (the usage's `CacheRead` plus `CacheWrite`, passed to `answered`) and the cache isn't cold (`cacheCold`, 5 minutes). Past 5 minutes it stays while the vendor said it keeps at least `cacheWorth` tokens of the conversation for an hour (`hourCache`, #2302): a reply with Anthropic's 1-hour writes (`cache_creation.ephemeral_1h_input_tokens`, the usage's `CacheWrite1h`, as Claude Code asks of a subscription) keeps what it read plus those for `cacheHour`, a later reply on the same account and model that reads all of it starts the hour again, and one that reads less (moved, compacted) drops it. The stick keeps it on disk too (`cacheHour`, `cacheHourAt`), and a group's rules keep the turn's effort the same hour (`keepsEffort`). Vendors that report no 1-hour write are unchanged. `TestAffinityKeepsAnHourCachePastFiveMinutes`, `TestHourCacheAfter`. What was written counts: a first turn reads nothing and writes the whole prompt (Anthropic's `cache_creation_input_tokens`), and the second turn stays to read it rather than moving as `no-cache` and building it again elsewhere (vilicvane on Discord). The stick keeps both, on disk too (`affinity.json`'s `cacheWrite`, absent in a file from before and read as 0, so such a stick is judged by its read alone), and the trace's `Affinity.CacheWrite` lets the Routing page say what was read and what was written. A vendor that keeps a prompt cached without reporting it (OpenAI's automatic prefix cache, Gemini) says neither of a turn that read none, so the prompt that turn was sent (the usage's `Input`) stands for the write and its second turn stays too; a reply that reports no usage at all counts as no cache, as before. `TestAffinityKeepsACacheJustWritten`, `TestAffinityKeepsACacheTheVendorKeptSilently`, `routing-affinity-cache-write.test.cjs`.
+- Routing decides who goes first only when nobody has answered the conversation yet. Affinity keeps the answerer while its cache is worth it. Auto mode keeps it across turns while the last reply read from the cache and wrote to it at least `cacheWorth` (1024) tokens between them (the usage's `CacheRead` plus `CacheWrite`, passed to `answered`) and the cache isn't cold (`cacheCold`, 5 minutes; for a ChatGPT account `chatGPTCold`, an hour, as measured on a real account (a thread left 7 to 60 minutes read back from cache all it had sent before), since the ChatGPT backend keeps a Codex conversation cached far longer and a move sends the whole context to the other account uncached: `coldAfter`, which a group's rules read too for `keepsEffort`). Past that window it stays while the vendor said it keeps at least `cacheWorth` tokens of the conversation for an hour (`hourCache`, #2302): a reply with Anthropic's 1-hour writes (`cache_creation.ephemeral_1h_input_tokens`, the usage's `CacheWrite1h`, as Claude Code asks of a subscription) keeps what it read plus those for `cacheHour`, a later reply on the same account and model that reads all of it starts the hour again, and one that reads less (moved, compacted) drops it. The stick keeps it on disk too (`cacheHour`, `cacheHourAt`), and a group's rules keep the turn's effort the same hour (`keepsEffort`). Vendors that report no 1-hour write are unchanged. `TestAffinityKeepsAnHourCachePastFiveMinutes`, `TestHourCacheAfter`. What was written counts: a first turn reads nothing and writes the whole prompt (Anthropic's `cache_creation_input_tokens`), and the second turn stays to read it rather than moving as `no-cache` and building it again elsewhere (vilicvane on Discord). The stick keeps both, on disk too (`affinity.json`'s `cacheWrite`, absent in a file from before and read as 0, so such a stick is judged by its read alone), and the trace's `Affinity.CacheWrite` lets the Routing page say what was read and what was written. A vendor that keeps a prompt cached without reporting it (OpenAI's automatic prefix cache, Gemini) says neither of a turn that read none, so the prompt that turn was sent (the usage's `Input`) stands for the write and its second turn stays too; a reply that reports no usage at all counts as no cache, as before. `TestAffinityKeepsACacheJustWritten`, `TestAffinityKeepsACacheTheVendorKeptSilently`, `routing-affinity-cache-write.test.cjs`.
- Strict (`strict`, #1797, miniers) never moves a conversation to another key or account (`who`). Its stick is kept as session's is, and while it is fresh `affine` keeps only the candidates of that `who` (`strictOnly`): its other models stay, so a group's rules still pick among them, but no other account, key or provider is asked, and a resting or spent one is asked anyway. A failure there reaches the agent and rests the account as a non-lone candidate's would (`strictAlone`), so new conversations go elsewhere. A reply that breaks off after its 200 (#733) forgets the stick under every other mode; under strict it doesn't, so the agent's retry goes back to the same account, member or key, failing there while it fails and going on there once it answers (imwhatiam123 on Discord: a group of Qoder CN and WorkBuddy AI moved a session to WorkBuddy AI, the retry routed as a new conversation; `TestStrictGroupKeepsItsMemberPastABrokenReply`). Before planning, `strictKeep` takes the stuck account back from what holds new conversations off it: a usage cap (`pl.capped`, `capHeldOnly`), so the conversation goes on past it. An account held for credits it may not spend is answered 429 naming the account and Strict, and nothing reaches the vendor, rather than moved. An account removed, switched off, barred, or not allowed for the calling key is gone, and the conversation moves as under session. The trace says `strict` even with one candidate. `strict_affinity_test.go`, `stays-strict.test.cjs`.
- In turn (`rotate`), a conversation nobody has answered yet goes to the key, account or model the fewest other conversations of the provider's model or the group answered in the last `heldFor` (30 minutes) are on (`leastHeld`, #946). Ties, resting ones and a group with a group in it keep routing's order. With session affinity, two sessions of one group so stay on different models.
- A ChatGPT account that holds credits answers past its windows on them; the vendor spends them. That is the default. Set not to spend them (Usage card's *Use credits*, `magpie quota credits <account> off`), the account is held at 100% of a window till it renews, by magpie's latest reading of its windows (`provider.Allowances`, refreshed about every minute), so the other accounts, groups and fallbacks take the request. A cap set on the account (1–99%) takes precedence. Unlike a cap, the hold is at the vendor's own 100%, so an account allowed to spend its resets spends one there, both when every candidate is held and after the last one is refused. A reset made forgets the account's cached allowance (`renewedNow`), so neither hold outlives the windows it started again. One let spend its credits, and holding some by its last `/wham/usage` (`credits.has_credits` with a balance over 0, or `unlimited`; `QuotaWindow.Credits`, `Limit.Credits`, `Allowance.OnCredits`), is not out when a window fills (豆子 and Zhenzhen on Discord): ChatGPT may answer the turn that uses the window up 429 `usage_limit_reached` before it spends the credits, and the next turn on them. So that 429 is asked again at once, once a request, on the same account, before a reset is looked at, on the routed path (`creditsAgain`, `creditsAsked`, `Try.Credits`) and on Codex's own sign-in's relay (`relayOnCredits`), rather than ended for the agent, which ended a running session while a new one worked. Refused again, it rests `fallbackCooldown` (`Rest.By` `credits`) and is read again, not until its window renews, and its full windows rest it nowhere else (`full` is zero for it, `spendsCredits`). Smart, Pace and Least used put such an account first of the spent tier (`routes.Seat.Credits`, `spentBefore`): after those with allowance left, ahead of those used up with nothing to go on with (`Weighed.Credits`, told on the Routing page). Switching to credits early, at a few percent left, was weighed and not done: the vendor spends credits only once the allowance is used up, so it would only waste what is left. `TestCodexCreditsAskedAgainAtTheEdge`, `TestCodexCreditsNoLongRest`, `TestCodexCreditsFirstOfTheSpent`, `TestCreditsFirstOfTheSpent`, `routing-codex-credits-again.test.cjs`.
diff --git a/internal/gateway/affinity.go b/internal/gateway/affinity.go
index 835691a1f..2cc1c0998 100644
--- a/internal/gateway/affinity.go
+++ b/internal/gateway/affinity.go
@@ -52,6 +52,17 @@ const (
// cacheCold is how long a vendor keeps a prompt cached without it being
// read: five minutes at Anthropic and at OpenAI, the shortest there is.
cacheCold = 5 * time.Minute
+ // chatGPTCold is how long the ChatGPT backend keeps a Codex
+ // conversation's prompt cached without it being read. Measured on a
+ // real account (10-27, gpt-5.6-terra), a Codex thread left 7, 12, 25,
+ // 45 and 60 minutes read back from cache all it had sent before (79–99%
+ // of each request), so it is
+ // held for the hour measured, not cacheCold's five minutes.
+ // Moving a conversation between two ChatGPT accounts after five
+ // minutes sent its whole context to the other uncached, and the
+ // reasoning the first had sealed was refused there (leelantao on X:
+ // two Plus accounts' allowance spent 90% in a day).
+ chatGPTCold = time.Hour
// cacheHour is how long Anthropic keeps what was written with
// cache_control's ttl "1h" (cache_creation's ephemeral_1h_input_tokens)
// without it being read, as Claude Code asks of a subscription (#2302)
@@ -62,6 +73,15 @@ const (
sticksKept = 512
)
+// coldAfter is how long p's vendor keeps a prompt cached without it being
+// read: cacheCold, or chatGPTCold for a ChatGPT account.
+func coldAfter(p provider.Provider) time.Duration {
+ if p.Account != nil && p.Account.Agent == "codex" {
+ return chatGPTCold
+ }
+ return cacheCold
+}
+
// Affinity is what the trace tells of a request's conversation and whether
// it stayed with who answered it last.
type Affinity struct {
@@ -536,8 +556,9 @@ func affine(scope, mode string, rotate bool, in http.Header, from provider.Proto
// what it wrote counts with what it read: the cache a first turn
// built is there for the next to read
a.Why = "no-cache"
- case time.Since(st.at) > cacheCold && !st.hour.warm(time.Now()):
- // past five minutes, kept still where the vendor said it
+ case time.Since(st.at) > coldAfter(cs[at].p) && !st.hour.warm(time.Now()):
+ // past what the vendor keeps unread (five minutes, longer at
+ // the ChatGPT backend), kept still where the vendor said it
// keeps enough for an hour (#2302)
a.Why = "cold"
default:
diff --git a/internal/gateway/chatgpt_cold_test.go b/internal/gateway/chatgpt_cold_test.go
new file mode 100644
index 000000000..5395068c8
--- /dev/null
+++ b/internal/gateway/chatgpt_cold_test.go
@@ -0,0 +1,83 @@
+package gateway
+
+import (
+ "net/http"
+ "testing"
+ "time"
+
+ "github.com/yetone/magpie/internal/provider"
+)
+
+// A conversation on one of two ChatGPT accounts stays on it across a
+// pause the ChatGPT backend still has it cached through (leelantao on X:
+// two Plus accounts spent 90% in a day). Five minutes was every vendor's
+// cold, so after a coffee break routing moved the conversation to the
+// other account, which read none of it from cache and refused the
+// reasoning the first had sealed. An API key's conversation is cold at
+// five minutes as before, and a ChatGPT one past chatGPTCold.
+func TestChatGPTConversationStaysPastFiveMinutes(t *testing.T) {
+ chatgpt := func(user string) candidate {
+ return candidate{p: provider.Provider{ID: "codex", Account: &provider.Account{Agent: "codex", User: user}}, model: "gpt-6-sol", rest: "codex"}
+ }
+ keyed := func(key string) candidate {
+ return candidate{p: provider.Provider{ID: "oai", Key: key, Responses: "https://api.openai.com/v1"}, model: "gpt-6-sol", rest: "oai"}
+ }
+ first := []byte(`{"input":[{"type":"message","role":"user","content":[{"type":"input_text","text":"fix the bug"}]}]}`)
+ again := []byte(`{"input":[{"type":"message","role":"user","content":[{"type":"input_text","text":"fix the bug"}]},{"type":"message","role":"assistant","content":[{"type":"output_text","text":"done"}]},{"type":"message","role":"user","content":[{"type":"input_text","text":"now the tests"}]}]}`)
+ for i, c := range []struct {
+ name string
+ last, next candidate
+ ago time.Duration
+ why string
+ }{
+ {"ChatGPT, 6 minutes on", chatgpt("[email protected]"), chatgpt("[email protected]"), 6 * time.Minute, "cache"},
+ {"ChatGPT, 12 minutes on", chatgpt("[email protected]"), chatgpt("[email protected]"), 12 * time.Minute, "cache"},
+ {"ChatGPT, past its window", chatgpt("[email protected]"), chatgpt("[email protected]"), chatGPTCold + time.Minute, "cold"},
+ {"an API key, 6 minutes on", keyed("sk-a"), keyed("sk-b"), 6 * time.Minute, "cold"},
+ {"an API key, 4 minutes on", keyed("sk-a"), keyed("sk-b"), 4 * time.Minute, "cache"},
+ } {
+ h := http.Header{}
+ h.Set("x-claude-code-session-id", "chatgpt-cold-"+string(rune('a'+i)))
+ // the other account goes first by routing: Smart puts the one with
+ // more left first, which after a turn on the last is the other
+ cs := []candidate{c.next, c.last}
+ pl := planned{order: []Weighed{{}, {}}}
+ _, _, _, key := affine("t", "", false, h, provider.Responses, first, first, cs, pl)
+ answered(key, c.last, 1, Usage{Input: 1200, CacheRead: 20000, CacheWrite: 1200})
+ t.Cleanup(func() { sticks.Lock(); delete(sticks.m, key); sticks.Unlock() })
+ sticks.Lock()
+ st := sticks.m[key]
+ st.at = st.at.Add(-c.ago)
+ sticks.m[key] = st
+ sticks.Unlock()
+ got, _, a, _ := affine("t", "", false, h, provider.Responses, again, again, cs, pl)
+ if a.Why != c.why || a.Kept != (c.why == "cache") {
+ t.Fatalf("%s: why %q kept %v, want %q", c.name, a.Why, a.Kept, c.why)
+ }
+ if want := c.next; c.why == "cache" {
+ want = c.last
+ if got[0].who() != want.who() {
+ t.Fatalf("%s: went to %s, want %s", c.name, got[0].who(), want.who())
+ }
+ } else if got[0].who() != want.who() {
+ t.Fatalf("%s: went to %s, want routing's %s", c.name, got[0].who(), want.who())
+ }
+ }
+}
+
+// A group's rules keep a ChatGPT conversation's effort through the same
+// pause (keepsEffort): a lower pick then reads none of the cache back.
+func TestChatGPTKeepsEffortPastFiveMinutes(t *testing.T) {
+ at := time.Date(2026, 10, 27, 9, 0, 0, 0, time.UTC)
+ tr := turnRule{effort: "high", at: at, answered: at, size: 40_000, cold: chatGPTCold}
+ if !keepsEffort(tr, "low", 41_000, at.Add(12*time.Minute)) {
+ t.Fatal("a ChatGPT conversation 12 minutes on dropped its effort")
+ }
+ if keepsEffort(tr, "low", 41_000, at.Add(chatGPTCold+time.Minute)) {
+ t.Fatal("kept its effort past the ChatGPT backend's window")
+ }
+ tr.cold = 0
+ if keepsEffort(tr, "low", 41_000, at.Add(6*time.Minute)) {
+ t.Fatal("another vendor's conversation kept its effort past five minutes")
+ }
+}
diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go
index b244270e4..7b67245cf 100644
--- a/internal/gateway/gateway.go
+++ b/internal/gateway/gateway.go
@@ -3730,10 +3730,10 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
// only it can read (#2271)
sealedBy = &c
if hit != nil && !hit.Compact {
- ruleAnswered(ruleAt, call.Usage)
+ ruleAnswered(ruleAt, call.Usage, coldAfter(c.p))
}
for _, at := range nestedAt {
- ruleAnswered(at, call.Usage)
+ ruleAnswered(at, call.Usage, coldAfter(c.p))
}
} else if hw.refused {
try.Fail = failRefused
diff --git a/internal/gateway/rules.go b/internal/gateway/rules.go
index c6356bba4..df2465704 100644
--- a/internal/gateway/rules.go
+++ b/internal/gateway/rules.go
@@ -117,6 +117,9 @@ type turnRule struct {
// hour is what the vendor said it keeps of the conversation for an
// hour (Anthropic's 1-hour cache writes), warm past cacheCold (#2302)
hour hourCache
+ // cold is how long the vendor that answered last keeps the
+ // conversation cached unread (coldAfter); 0 is cacheCold
+ cold time.Duration
}
var turnRules = struct {
@@ -419,7 +422,7 @@ func alsoIDs(m provider.Member) []string {
// ruleAnswered keeps what the vendor counted a conversation's request as,
// for its next turn's rules to go by.
-func ruleAnswered(key string, u Usage) {
+func ruleAnswered(key string, u Usage, cold time.Duration) {
n := u.Input + u.CacheRead + u.CacheWrite
if n <= 0 {
return
@@ -428,7 +431,7 @@ func ruleAnswered(key string, u Usage) {
defer turnRules.Unlock()
if tr, ok := turnRules.m[key]; ok {
now := ruleClock()
- tr.input, tr.answered, tr.hour = n, now, tr.hour.after(u, now)
+ tr.input, tr.answered, tr.hour, tr.cold = n, now, tr.hour.after(u, now), cold
turnRules.m[key] = tr
}
}
@@ -442,7 +445,8 @@ func ruleAnswered(key string, u Usage) {
// while the cache is warm the effort only goes up — a hard turn still gets
// the reasoning it needs, and a conversation pays for at most one miss per
// level — and a lower pick waits for a turn the cache is lost to anyway:
-// one after cacheCold without an answer (an hour, where the vendor said it
+// one after cacheCold without an answer (tr.cold at the ChatGPT backend,
+// which keeps it longer; an hour, where the vendor said it
// keeps the conversation that long: tr.hour, #2302), or after the agent
// compacted the conversation (it is now under half as long as its last
// request).
@@ -457,7 +461,11 @@ func keepsEffort(tr turnRule, pick string, tokens int, now time.Time) bool {
if tr.answered.After(last) {
last = tr.answered
}
- if now.Sub(last) > cacheCold && !tr.hour.warm(now) {
+ cold := tr.cold
+ if cold <= 0 {
+ cold = cacheCold
+ }
+ if now.Sub(last) > cold && !tr.hour.warm(now) {
return false
}
if tr.size > 0 && tokens < tr.size/2 {