Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
gateway: a relayed Chat stream's tool call with no index is given its call's (#3386)
JoeyWu-techba4b754b22e2
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex 7a49a128e..5ad40832b 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -149,6 +149,7 @@ recorded as a prediction failure (`noPrediction`).- A request built for a Messages upstream (`buildAnthropic`: a Chat, Responses or Gemini client translated) has its tool exchanges mended after its messages are built (`pairToolBlocks`, #1558, ggbdpq), since Anthropic answers 400 for a `tool_use` the next message doesn't answer and for a `tool_result` naming no call of the message before it. A call left unanswered gets the synthetic interrupted result, `is_error` set, first in the user message after it or in one of its own; a result that answers no call goes as a text block with what it said (its images as they came); a second answer to one call is dropped. A well-formed exchange goes as it was built. An Anthropic client relayed to a Messages upstream isn't mended. [`anthropic_tool_pairing_test.go`](../../internal/gateway/anthropic_tool_pairing_test.go).- A Responses request bound for anything but a ChatGPT sign-in has its tool exchanges mended in `attempt`, before it is relayed or translated, plugins included ([`responses.go`](../../internal/gateway/responses.go)). An item inside a run of calls and results (after one of the run's results and before the last, or before any result when the model didn't write it) moves to the end of that run (`joinToolRuns`, #1752, Edward4226): Codex writes its `<image_resize_notice>` right after each result whose image it resized, so a batch of two calls had it between their results, and a strict upstream answered 400 (OpenCode Go/DeepSeek "No tool output found for tool call …", Volcengine's "must be followed by tool messages", #1822); a Chat translation answered the second call with a synthetic result. An item carrying a `call_id` never moves, the model's own reasoning between its calls stays, and its text between them (an assistant message, with the reasoning right before it) goes ahead of the run's first call, so the step's calls stand together: OpenCode Go opens a new assistant message at such text and answered the call before it "No tool output found for tool call …" (#2100). A call opens a run only when a result after it answers it. A result with no call id becomes a user message after its run (`orphanedToolOutputs`), and a call nothing answers gets a synthetic result at the end of its run (`pairToolItems`, #1341). `TestRelayedCodexImageBatchKeepsItsResultsTogether`, `TestJoinToolRunsOnlyMovesWhatBreaksARun`, `TestRelayedTextBetweenABatchsCallsIsAnswered`.- A Chat request has every tool call that came with no id (`""`, null or none) given one in `attempt`, before it is relayed or translated, plugins included (`chatCallIDs` in [`chat_call_ids.go`](../../internal/gateway/chat_call_ids.go), #2611, RunMintOn). An upstream that sometimes streams a call without an id (OpenCode Zen's free DeepSeek) has the agent keep it as `""`, which that upstream takes back, while WorkBuddy AI turns a streamed request carrying one away (400, code 11133) and Anthropic takes no empty `tool_use` id. The id is made from the call's place in the request, its name and its arguments, never at random, so the next turn's history goes byte for byte and the upstream's prompt cache holds; the tool messages after the call that answer with no `tool_call_id` take the made ids in the calls' order. A body whose calls all have ids goes as it came. On the reply side, a relayed Chat stream's call whose first fragment has no id is given one, and a later fragment's empty id is left out (`chatTidy`); a whole reply's call is given one too (`chatWhole`). `TestEmptyCallIDRelayedWithOne`, `TestEmptyCallIDReachesPluginWithOne`, `TestEmptyCallIDTranslatedWithOne`, `TestStreamedCallGivenAnID` in [`chat_call_ids_test.go`](../../internal/gateway/chat_call_ids_test.go).+- A relayed Chat stream's tool call fragment with no `index` (none, or null) is given its call's (`streamCallIndexes` in [`chattidy.go`](../../internal/gateway/chattidy.go), #3386). Gemini's OpenAI-compatible API sends its calls with none, where the OpenAI contract has one on every fragment: the AI SDK's OpenAI provider (Cursor's Private Inference) turned the chunk away ("expected number, received undefined" at `tool_calls[0].index`), and OpenAI's node SDK's stream helper kept no call at all. A fragment with an id other than the open call's, or after the first in its delta, is the next call; any other goes on with the open one. So two calls in chunks of their own are 0 and 1, never both 0, which that helper merges into one call. A fragment with an index keeps it. `TestChatTidyGivesCallsAnIndex` in [`chattidy_test.go`](../../internal/gateway/chattidy_test.go), `TestGeminiCompatRelayedCallsHaveAnIndex` in [`gemini_compat_cut_test.go`](../../internal/gateway/gemini_compat_cut_test.go).- A relayed Chat reply whose choice made calls and ended `stop` ends `tool_calls`, streamed (`chatTidy`) and whole (`chatWhole`), as the translations' collector already ends it (`collector.finish`): told it stopped, a client that reads `finish_reason` ends its turn with the calls not run (喜欢一个人 on Discord, a relay in front of deepseek-v4.1-flash). Any other reason stays as it came. `TestChatCallsEndToolCalls` in [`chattidy_finish_test.go`](../../internal/gateway/chattidy_finish_test.go).- Codex's model list from the backend path (`codexModels`) is the ChatGPT backend's for its sign-in and then magpie's (`provider.CodexListed`), its ETag and every reply's `X-Models-Etag` tagged with `provider.CodexListTag` so a change has Codex ask again. A Codex there only for account failover, not connected (`CodexOwnOnly`, `agent.CodexOwnOnly`; #1385), gets the backend's alone, tagged with `provider.CodexOwnListTag`; neither tag contains the other, so connecting or disconnecting has Codex fetch the list again. The user's picks on the codex provider (`provider.CodexNativePicked`) narrow the backend's listed models. They keep its hidden ones (`visibility: "hide"`), which no picker shows. One of those is `codex-auto-review`: when the list lacks it and no reviewer is set, Codex's auto-review runs on the conversation's model at low effort (#1460, `TestCodexModelListPicksKeepHidden`). The account's own models kept out of agents' lists for routing groups (`provider.CodexNativeUnlisted`) are dropped from it too, hidden ones kept: every one when the codex provider is "Only through routing groups" (`Provider.Unlisted`), and those a group that unlists its members names (`Group.Unlists`). A Codex there only for failover keeps them, as it is handed no groups; `spawnChoices` drops the same ones (#1602, `TestCodexModelListUnlistedForGroups`).- Reasoning state goes through magpie's translation as the vendor wrote it when the agent and the upstream speak the same API (#1445, XYenon): a non-streaming Messages request to a stream-only account, web search on the agent's own API, a plugin, failover. The IR carries it sealed (`Part.Sealed`, `SealedBy`; events `KThinkStart`, `KSealed`, `KTextSig` in [`ir.go`](../../internal/gateway/ir.go)), so each thinking block or reasoning item stays apart, whole or streamed, and a seal is written only for its own API (`sealedBy`). Anthropic's `redacted_thinking` goes back raw (to DeepSeek as a thinking block, below), and a thinking block signed with no text keeps its signature (`aBlock` still writes `"thinking": ""`, #1447), on Claude's subscription too. A Responses reasoning item keeps its id, summary and `encrypted_content`, and goes back to a Responses upstream as it came when it has no reasoning text (`sealedReasoning`; text, another vendor's, is still left out for OpenAI, #1411). Gemini's `thoughtSignature` stays on the call it signed and on text; a later call of a signed step goes without one, and only an unsigned step or Claude on Antigravity gets `skip_thought_signature_validator`. Another API's agent gets the call's signature in its id, as on chat (#687), with its length in the mark (`__ts<n>__`), so an id a client shortened carries no signature and the call goes as unsigned (#1536); it gets no one else's seal. The signature itself is kept in magpie's folder, `gemini-signatures/<key>`, named by its hash (`keepSignature` in [`gemini_sigstore.go`](../../internal/gateway/gemini_sigstore.go)), and the id carries only the key (`__tsk16__<key>`), so an id stays under 64 characters however long the step's signature (#2411: LobeHub's PostgreSQL index turned away a 3KB id). A key magpie no longer has, or whose file doesn't hash to it, reads as no signature; one not read for 90 days is removed. Where the folder can't be written, the id carries the signature as before. Carrying reasoning across APIs (an envelope) isn't done.diff --git a/internal/gateway/chattidy.go b/internal/gateway/chattidy.goindex 684320165..24a7c5624 100644--- a/internal/gateway/chattidy.go+++ b/internal/gateway/chattidy.go@@ -2,6 +2,7 @@ package gatewayimport ("bytes"+ "cmp""encoding/json""slices""strings"@@ -52,11 +53,20 @@ import (// the calls not run (喜欢一个人 on Discord, a relay in front of// deepseek-v4.1-flash), as the translations' collector already has it.//+// And Gemini's OpenAI-compatible API sends its tool calls with no "index"+// (Linka's frames in gemini_compat_cut_test.go), where the contract has+// one on every fragment. The AI SDK's OpenAI provider, which Cursor's+// Private Inference reads Chat with, turns such a chunk away ("expected+// number, received undefined" at tool_calls[0].index), and the call is+// never made (#3386). A fragment with none is given its call's: a new id,+// or a later place in its delta, is the next call.+//// Lines pass whole, as they came, unless one has any of them.type chatTidy struct {- buf []byte- ids streamCallIDs- called map[int]bool // the choices that made a call+ buf []byte+ ids streamCallIDs+ indexes streamCallIndexes+ called map[int]bool // the choices that made a call}var (@@ -167,6 +177,13 @@ func (t *chatTidy) line(line []byte) []byte {delete(delta, "tool_calls")inDelta = true} else if ok {+ if t.indexes == nil {+ t.indexes = streamCallIndexes{}+ }+ if given, chg := t.indexes.give(choice, calls); chg {+ calls, delta["tool_calls"] = given, given+ inDelta = true+ }if t.ids == nil {t.ids = streamCallIDs{}}@@ -235,6 +252,64 @@ func withoutEmptyNames(raw json.RawMessage) (json.RawMessage, bool) {return out, true}+// streamCallIndexes is what chatTidy keeps to give a streamed tool call+// the upstream sent no index for one: each choice's open call.+type streamCallIndexes map[int]*openCall++// openCall is a choice's open tool call: its id, the index it goes by,+// and the next index to give.+type openCall struct {+ open bool+ id string+ index, next int+}++// give gives each fragment of a delta's tool_calls with no index (none,+// or null) the index of the call it belongs to, and reports whether it+// gave any. A fragment with an id other than the open call's, or after+// the first in its delta, starts the next call, as chatDecoder reads a+// call with a new id; any other goes on with the open one. So two calls+// in chunks of their own are 0 and 1: given both 0, by their place, OpenAI's+// node SDK merged them into one call. A fragment with an index keeps it,+// and the next one given comes after it.+func (s streamCallIndexes) give(choice int, raw json.RawMessage) (json.RawMessage, bool) {+ var calls []map[string]json.RawMessage+ if decodeJSON(raw, &calls) != nil {+ return raw, false+ }+ o := s[choice]+ if o == nil {+ o = &openCall{}+ s[choice] = o+ }+ changed := false+ for k, c := range calls {+ id := callID(c["id"])+ if v, ok := c["index"]; ok && string(v) != "null" {+ var n int+ if decodeJSON(v, &n) == nil {+ o.open, o.id, o.index = true, cmp.Or(id, o.id), n+ o.next = max(o.next, n+1)+ }+ continue+ }+ if !o.open || k > 0 || id != "" && id != o.id {+ o.open, o.id, o.index = true, id, o.next+ o.next+++ }+ c["index"], _ = marshalPlain(o.index)+ changed = true+ }+ if !changed {+ return raw, false+ }+ out, err := json.Marshal(calls)+ if err != nil {+ return raw, false+ }+ return out, true+}+// partsText reads a content that is an array of typed parts: its text,// and the text of its thinking parts. ok is false for any other content,// a string or null.diff --git a/internal/gateway/chattidy_test.go b/internal/gateway/chattidy_test.goindex 8ebe8f6ee..a4de323ef 100644--- a/internal/gateway/chattidy_test.go+++ b/internal/gateway/chattidy_test.go@@ -1,8 +1,10 @@package gatewayimport (+ "bytes""encoding/json""net/http/httptest"+ "reflect""strings""testing"@@ -150,3 +152,56 @@ func TestChatTidyToolCallNames(t *testing.T) {t.Fatalf("got %q", out)}}++// A streamed call with no index is given its call's (#3386): one index for+// a call's fragments, the next for the next call (a new id, or a later+// place in its delta), each choice its own, a null index none. A stream+// whose calls have their index goes as it came.+func TestChatTidyGivesCallsAnIndex(t *testing.T) {+ var tidy chatTidy+ for i, c := range []struct {+ choices string+ want [][]float64 // each choice's calls' indexes+ }{+ {`[{"index":0,"delta":{"tool_calls":[{"id":"a","type":"function","function":{"name":"grep","arguments":""}}]}}]`, [][]float64{{0}}},+ {`[{"index":0,"delta":{"tool_calls":[{"function":{"arguments":"{\"pat"}}]}}]`, [][]float64{{0}}},+ {`[{"index":1,"delta":{"tool_calls":[{"id":"x","type":"function","function":{"name":"ls","arguments":"{}"}}]}}]`, [][]float64{{0}}},+ {`[{"index":0,"delta":{"tool_calls":[{"id":"a","function":{"arguments":"tern\"}"}}]}}]`, [][]float64{{0}}},+ {`[{"index":0,"delta":{"tool_calls":[{"id":"b","type":"function","function":{"name":"ls","arguments":"{}"}},{"type":"function","function":{"name":"cat","arguments":"{"}}]}}]`, [][]float64{{1, 2}}},+ {`[{"index":0,"delta":{"tool_calls":[{"index":null,"function":{"arguments":"}"}}]}}]`, [][]float64{{2}}},+ {`[{"index":0,"delta":{"tool_calls":[{"id":"d","type":"function","function":{"name":"pwd","arguments":"{}"}}]}},{"index":1,"delta":{"tool_calls":[{"id":"y","type":"function","function":{"name":"pwd","arguments":"{}"}}]}}]`, [][]float64{{3}, {1}}},+ } {+ out := tidy.write([]byte(`data: {"choices":` + c.choices + "}\n"))+ var chunk struct {+ Choices []struct {+ Delta struct {+ ToolCalls []map[string]any `json:"tool_calls"`+ } `json:"delta"`+ } `json:"choices"`+ }+ if err := json.Unmarshal(bytes.TrimPrefix(bytes.TrimSpace(out), []byte("data:")), &chunk); err != nil {+ t.Fatalf("line %d unreadable %q: %v", i, out, err)+ }+ var got [][]float64+ for _, ch := range chunk.Choices {+ var idx []float64+ for _, tc := range ch.Delta.ToolCalls {+ n, ok := tc["index"].(float64)+ if !ok {+ n = -1+ }+ idx = append(idx, n)+ }+ got = append(got, idx)+ }+ if !reflect.DeepEqual(got, c.want) {+ t.Errorf("line %d: indexes %v, want %v: %s", i, got, c.want, out)+ }+ }++ indexed := `data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_1","type":"function","function":{"name":"ls","arguments":"{}"}},{"index":1,"id":"call_2","type":"function","function":{"name":"pwd","arguments":"{}"}}]}}]}` + "\n"+ var other chatTidy+ if out := string(other.write([]byte(indexed))); out != indexed {+ t.Fatalf("a stream with its indexes changed: %q", out)+ }+}diff --git a/internal/gateway/gemini_compat_cut_test.go b/internal/gateway/gemini_compat_cut_test.goindex be7a25e47..681d6094a 100644--- a/internal/gateway/gemini_compat_cut_test.go+++ b/internal/gateway/gemini_compat_cut_test.go@@ -159,3 +159,53 @@ func TestGeminiCompatFinishAfterPartialLine(t *testing.T) {t.Errorf("the finish_reason began inside the line still pending: %q", body)}}++// Gemini's OpenAI-compatible API sends a call with no index (Linka's+// frames), where the OpenAI contract has one on every fragment. The AI+// SDK's OpenAI provider, which Cursor's Private Inference reads Chat with,+// turned the chunk away ("expected number, received undefined" at+// tool_calls[0].index), and the call was never made (#3386). Relayed,+// every call has its index: Linka's call 0, a second call in a chunk of+// its own 1, and #3386's chunk, with no id either, 0 and an id.+func TestGeminiCompatRelayedCallsHaveAnIndex(t *testing.T) {+ const ask = `{"model":"MODEL","stream":true,"messages":[{"role":"user","content":"hi"}]}`+ second := strings.NewReplacer(`"call_1"`, `"call_2"`, `run()`, `two()`).Replace(compatCall)+ reported := `data: {"choices":[{"delta":{"role":"assistant","tool_calls":[{"extra_content":{"google":{"thought_signature":"c2ln"}},"function":{"arguments":"{\"code\":\"run()\"}","name":"codemode"}}]},"index":0}],"model":"models/gemini-4-argon","object":"chat.completion.chunk"}` + "\n\n"+ for _, c := range []struct {+ name, reply string+ ids []string // how each call's id starts+ }{+ {"Linka's call", compatCall + compatStop + compatDone, []string{"call_1"}},+ {"a second call", compatCall + second + compatStop + compatDone, []string{"call_1", "call_2"}},+ {"#3386's chunk", reported + compatStop + compatDone, []string{"call_"}},+ } {+ t.Run(c.name, func(t *testing.T) {+ code, body := compatServer(t, "gemini-4-argon", c.reply, "/v1/chat/completions", ask)+ if code != 200 {+ t.Fatalf("status %d: %s", code, body)+ }+ var calls []map[string]any+ for _, ev := range events(body) {+ chs, _ := ev["choices"].([]any)+ for _, ch := range chs {+ d, _ := ch.(map[string]any)["delta"].(map[string]any)+ tcs, _ := d["tool_calls"].([]any)+ for _, tc := range tcs {+ calls = append(calls, tc.(map[string]any))+ }+ }+ }+ if len(calls) != len(c.ids) {+ t.Fatalf("%d calls, want %d: %s", len(calls), len(c.ids), body)+ }+ for i, call := range calls {+ if n, ok := call["index"].(float64); !ok || n != float64(i) {+ t.Errorf("call %d went with index %v, want %d: %s", i, call["index"], i, body)+ }+ if id, _ := call["id"].(string); !strings.HasPrefix(id, c.ids[i]) {+ t.Errorf("call %d went with id %q, want one starting %q", i, id, c.ids[i])+ }+ }+ })+ }+}