Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

gateway: a Chat stream's tool calls sent by turns are read by index, one call each

yetone865df247278b
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index 649578549..dd2f86c1f 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -200,6 +200,7 @@ recorded as a prediction failure (`noPrediction`).
- A Chat client's own body relayed to a Chat upstream (`passthrough`) has its tool exchanges mended when they don't pair (`toolPairsOut` in [`chattidy.go`](../../internal/gateway/chattidy.go)), by the rule a translated one is built with (`pairToolMessages` in [`chat.go`](../../internal/gateway/chat.go)): Kimi's Chat API, like OpenAI's, turns the whole request away for an assistant's call the tool messages right after it don't answer ("an assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'…"), a user message between a call and its answer included, and for a tool message with no call before it ("tool_call_id is not found"), both checked against api.kimi.com/coding on 2026-10-26. Qoder's chat forked at an earlier turn, its model switched to Kimi Code, was refused on every prompt (Paidaixinbao on X). A turn's calls split over consecutive assistant messages join into one first (`joinSplitChatCalls`); a call left unanswered gets the synthetic interrupted result; an answer to no call becomes the user's text, a parts content by its text parts. A body whose exchange pairs goes byte for byte, answers out of the calls' order included, which Kimi takes (`TestChatRelayPairsToolsForKimi`, `TestChatRelayPairedToolsUntouched`).
- A Chat request has every tool call that came with no id (`""`, null or none) given one in `attempt`, before it is relayed or translated, plugins included (`chatCallIDs` in [`chat_call_ids.go`](../../internal/gateway/chat_call_ids.go), #2611, RunMintOn). An upstream that sometimes streams a call without an id (OpenCode Zen's free DeepSeek) has the agent keep it as `""`, which that upstream takes back, while WorkBuddy AI turns a streamed request carrying one away (400, code 11133) and Anthropic takes no empty `tool_use` id. The id is made from the call's place in the request, its name and its arguments, never at random, so the next turn's history goes byte for byte and the upstream's prompt cache holds; the tool messages after the call that answer with no `tool_call_id` take the made ids in the calls' order. A body whose calls all have ids goes as it came. On the reply side, a relayed Chat stream's call whose first fragment has no id is given one, and a later fragment's empty id is left out (`chatTidy`); a whole reply's call is given one too (`chatWhole`). `TestEmptyCallIDRelayedWithOne`, `TestEmptyCallIDReachesPluginWithOne`, `TestEmptyCallIDTranslatedWithOne`, `TestStreamedCallGivenAnID` in [`chat_call_ids_test.go`](../../internal/gateway/chat_call_ids_test.go).
- A relayed Chat stream's tool call fragment with no `index` (none, or null) is given its call's (`streamCallIndexes` in [`chattidy.go`](../../internal/gateway/chattidy.go), #3386). Gemini's OpenAI-compatible API sends its calls with none, where the OpenAI contract has one on every fragment: the AI SDK's OpenAI provider (Cursor's Private Inference) turned the chunk away ("expected number, received undefined" at `tool_calls[0].index`), and OpenAI's node SDK's stream helper kept no call at all. A fragment with an id other than the open call's, or after the first in its delta, is the next call; any other goes on with the open one. So two calls in chunks of their own are 0 and 1, never both 0, which that helper merges into one call. A fragment with an index keeps it. `TestChatTidyGivesCallsAnIndex` in [`chattidy_test.go`](../../internal/gateway/chattidy_test.go), `TestGeminiCompatRelayedCallsHaveAnIndex` in [`gemini_compat_cut_test.go`](../../internal/gateway/gemini_compat_cut_test.go).
+- A translated Chat stream's tool calls are read by `index`, not as one after another (`chatDecoder.toolCall` in [`chat.go`](../../internal/gateway/chat.go)). An upstream may send several calls' fragments by turns (index 0, 1, 0, 1…), the first of each with its id and name; read as a new call at each change of index, two calls reached Codex as six `function_call` items, the first two with `{}` and four with no name (mysterious_dove on Discord). The open call streams as it comes; a call that starts while the open one's arguments aren't yet whole JSON is held by its index and given whole, in index order, at `finish_reason` or `[DONE]`. Every translation from Chat (Responses, Messages, Gemini, a whole reply) reads the stream through it. `TestInterleavedToolCallsToResponses`, `TestInterleavedToolCallsToAnthropic`, `TestInterleavedToolCallsWhole`, `TestToolCallsNamedFirstThenArgued` in [`translate_stream_test.go`](../../internal/gateway/translate_stream_test.go).
- A relayed Chat reply whose choice made calls and ended `stop` ends `tool_calls`, streamed (`chatTidy`) and whole (`chatWhole`), as the translations' collector already ends it (`collector.finish`): told it stopped, a client that reads `finish_reason` ends its turn with the calls not run (喜欢一个人 on Discord, a relay in front of deepseek-v4.1-flash). Any other reason stays as it came. `TestChatCallsEndToolCalls` in [`chattidy_finish_test.go`](../../internal/gateway/chattidy_finish_test.go).
- Codex's model list from the backend path (`codexModels`) is the ChatGPT backend's for its sign-in and then magpie's (`provider.CodexListed`), its ETag and every reply's `X-Models-Etag` tagged with `provider.CodexListTag` so a change has Codex ask again. A Codex there only for account failover, not connected (`CodexOwnOnly`, `agent.CodexOwnOnly`; #1385), gets the backend's alone, tagged with `provider.CodexOwnListTag`; neither tag contains the other, so connecting or disconnecting has Codex fetch the list again. The user's picks on the codex provider (`provider.CodexNativePicked`) narrow the backend's listed models. They keep its hidden ones (`visibility: "hide"`), which no picker shows. One of those is `codex-auto-review`: when the list lacks it and no reviewer is set, Codex's auto-review runs on the conversation's model at low effort (#1460, `TestCodexModelListPicksKeepHidden`). The account's own models kept out of agents' lists for routing groups (`provider.CodexNativeUnlisted`) are dropped from it too, hidden ones kept: every one when the codex provider is "Only through routing groups" (`Provider.Unlisted`), and those a group that unlists its members names (`Group.Unlists`). A Codex there only for failover keeps them, as it is handed no groups; `spawnChoices` drops the same ones (#1602, `TestCodexModelListUnlistedForGroups`).
- Reasoning state goes through magpie's translation as the vendor wrote it when the agent and the upstream speak the same API (#1445, XYenon): a non-streaming Messages request to a stream-only account, web search on the agent's own API, a plugin, failover. The IR carries it sealed (`Part.Sealed`, `SealedBy`; events `KThinkStart`, `KSealed`, `KTextSig` in [`ir.go`](../../internal/gateway/ir.go)), so each thinking block or reasoning item stays apart, whole or streamed, and a seal is written only for its own API (`sealedBy`). Anthropic's `redacted_thinking` goes back raw (to DeepSeek as a thinking block, below), and a thinking block signed with no text keeps its signature (`aBlock` still writes `"thinking": ""`, #1447), on Claude's subscription too. A Responses reasoning item keeps its id, summary and `encrypted_content`, and goes back to a Responses upstream as it came when it has no reasoning text (`sealedReasoning`; text, another vendor's, is still left out for OpenAI, #1411). Gemini's `thoughtSignature` stays on the call it signed and on text; a later call of a signed step goes without one, and only an unsigned step or Claude on Antigravity gets `skip_thought_signature_validator`. Another API's agent gets the call's signature in its id, as on chat (#687), with its length in the mark (`__ts<n>__`), so an id a client shortened carries no signature and the call goes as unsigned (#1536); it gets no one else's seal. The signature itself is kept in magpie's folder, `gemini-signatures/<key>`, named by its hash (`keepSignature` in [`gemini_sigstore.go`](../../internal/gateway/gemini_sigstore.go)), and the id carries only the key (`__tsk16__<key>`), so an id stays under 64 characters however long the step's signature (#2411: LobeHub's PostgreSQL index turned away a 3KB id). A key magpie no longer has, or whose file doesn't hash to it, reads as no signature; one not read for 90 days is removed. Where the folder can't be written, the id carries the signature as before. Carrying reasoning across APIs (an envelope) isn't done.
diff --git a/internal/gateway/chat.go b/internal/gateway/chat.go
index 004ce7d07..da6dfe7b3 100644
--- a/internal/gateway/chat.go
+++ b/internal/gateway/chat.go
@@ -1058,11 +1058,22 @@ func (u Usage) chat() map[string]any {
// chatDecoder turns a Chat Completions stream into events. Tool calls come
// as indexed fragments; it tracks which one is open.
+//
+// Events give a call's arguments to the call started last, but Chat may
+// send the fragments of several calls by turns (index 0, 1, 0, 1…): read
+// as one call after another, two calls came out as six, the last four
+// with no name and pieces of the arguments (mysterious_dove). So the open
+// call streams as it comes, and a call that starts while the open one's
+// arguments aren't yet whole JSON is held, by its index, and given whole,
+// in index order, when the reply ends.
type chatDecoder struct {
- started bool
- tool int // index of the open tool call, -1 for none
- toolID string // id of the open tool call, as some relays repeat it on every fragment
- choice string // index of the first choice seen; an empty string means none yet
+ started bool
+ tool int // index of the open tool call, -1 for none
+ toolID string // id of the open tool call, as some relays repeat it on every fragment
+ toolWhole jsonWhole // where the open call's arguments stand in their JSON
+ toolDone bool // they are whole JSON: a call that starts now comes after it
+ held []*heldCall
+ choice string // index of the first choice seen; an empty string means none yet
// Gemini's OpenAI-compatible API, asked for thoughts, may give them
// in the text as a leading <thought>…</thought>, and MiniMax's Chat
// API (#1267), like the open models served without a reasoning
@@ -1261,6 +1272,7 @@ func (d *chatDecoder) end(emit func(Event)) {
func (d *chatDecoder) decode(data string, emit func(Event)) error {
if strings.TrimSpace(data) == "[DONE]" {
+ d.giveHeld(emit)
return nil
}
var ch struct {
@@ -1361,23 +1373,11 @@ func (d *chatDecoder) decode(data string, emit func(Event)) error {
if tc.Index != nil {
idx = *tc.Index
}
- // A delta carrying the id the open call already goes by — some
- // relays repeat it on every fragment, where the spec sends it
- // only on the first — or one more fragment of the open index,
- // continues that call; only a new id or a new index starts the
- // next one.
- if idx != d.tool || (tc.ID != "" && tc.ID != d.toolID) {
- d.tool, d.toolID = idx, tc.ID
- // a call Gemini signed goes to the client with an id
- // carrying the signature, which comes back with it
- emit(Event{Kind: KToolStart, ID: signedID(tc.ID, googleSignature(tc.ExtraContent)), Name: tc.Function.Name})
- }
- if tc.Function.Arguments != "" {
- emit(Event{Kind: KToolArgs, Text: tc.Function.Arguments})
- }
+ d.toolCall(idx, tc, emit)
}
if c.FinishReason != "" {
d.end(emit)
+ d.giveHeld(emit)
emit(Event{Kind: KStop, Stop: stopFromChat(c.FinishReason)})
}
}
@@ -1387,6 +1387,74 @@ func (d *chatDecoder) decode(data string, emit func(Event)) error {
return nil
}
+// heldCall is a tool call whose fragments came by turns with another's,
+// held until the reply ends.
+type heldCall struct {
+ index int
+ id, sent string // the id the upstream gave, and the one the client is given
+ name string
+ args strings.Builder
+}
+
+// toolCall reads one fragment of a tool call, the call at index idx. A
+// fragment carrying the id its call already goes by (some relays repeat it
+// on every fragment, where the spec sends it only on the first) or none
+// continues the latest call of its index; a new id or a new index starts
+// the next one.
+func (d *chatDecoder) toolCall(idx int, tc cToolCall, emit func(Event)) {
+ args := tc.Function.Arguments
+ newID := func(id string) bool { return tc.ID != "" && tc.ID != id }
+ for _, h := range slices.Backward(d.held) {
+ if h.index == idx {
+ if newID(h.id) {
+ break
+ }
+ h.args.WriteString(args)
+ return
+ }
+ }
+ if d.tool >= 0 && idx == d.tool && !newID(d.toolID) {
+ d.toolArgs(args, emit)
+ return
+ }
+ // a call Gemini signed goes to the client with an id carrying the
+ // signature, which comes back with it
+ sent := signedID(tc.ID, googleSignature(tc.ExtraContent))
+ if d.tool >= 0 && (!d.toolDone || len(d.held) > 0) {
+ h := &heldCall{index: idx, id: tc.ID, sent: sent, name: tc.Function.Name}
+ h.args.WriteString(args)
+ d.held = append(d.held, h)
+ return
+ }
+ d.tool, d.toolID, d.toolWhole, d.toolDone = idx, tc.ID, jsonWhole{}, false
+ emit(Event{Kind: KToolStart, ID: sent, Name: tc.Function.Name})
+ d.toolArgs(args, emit)
+}
+
+// toolArgs gives a piece of the open call's arguments.
+func (d *chatDecoder) toolArgs(args string, emit func(Event)) {
+ if args == "" {
+ return
+ }
+ d.toolDone = d.toolWhole.feed(args)
+ emit(Event{Kind: KToolArgs, Text: args})
+}
+
+// giveHeld gives the calls held, each whole, in index order.
+func (d *chatDecoder) giveHeld(emit func(Event)) {
+ if len(d.held) == 0 {
+ return
+ }
+ slices.SortStableFunc(d.held, func(a, b *heldCall) int { return a.index - b.index })
+ for _, h := range d.held {
+ emit(Event{Kind: KToolStart, ID: h.sent, Name: h.name})
+ if h.args.Len() > 0 {
+ emit(Event{Kind: KToolArgs, Text: h.args.String()})
+ }
+ }
+ d.held, d.tool, d.toolID, d.toolWhole, d.toolDone = nil, -1, "", jsonWhole{}, false
+}
+
// chatChoiceIndex accepts numeric indexes and numeric strings without
// changing how a missing or unreadable index is handled. JSON numbers such
// as 1.0 and 1 are the same choice.
diff --git a/internal/gateway/translate_stream_test.go b/internal/gateway/translate_stream_test.go
index 3cdeff3e0..243afec9a 100644
--- a/internal/gateway/translate_stream_test.go
+++ b/internal/gateway/translate_stream_test.go
@@ -386,3 +386,162 @@ func TestRepeatedToolCallIDKeepsParallelCalls(t *testing.T) {
t.Fatalf("calls = %v", calls)
}
}
+
+// interleavedCalls is a Chat stream whose two tool calls come a fragment
+// at a time, by turns: index 0, 1, 0, 1, 0, 1, the first fragment of each
+// with its id and name, the rest only pieces of the arguments
+// (mysterious_dove on Discord: Codex got six function_call items, the first
+// two with arguments {}, the other four with no name and half of them).
+var interleavedCalls = []string{
+ `{"id":"x","model":"m","choices":[{"index":0,"delta":{"role":"assistant","tool_calls":[{"index":0,"id":"call_read","type":"function","function":{"name":"read_file","arguments":"{\"pa"}}]},"finish_reason":null}]}`,
+ `{"id":"x","model":"m","choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":"call_list","type":"function","function":{"name":"list_dir","arguments":"{\"di"}}]},"finish_reason":null}]}`,
+ `{"id":"x","model":"m","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"th\":\"a/b"}}]},"finish_reason":null}]}`,
+ `{"id":"x","model":"m","choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"function":{"arguments":"r\":\"src"}}]},"finish_reason":null}]}`,
+ `{"id":"x","model":"m","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":".go\"}"}}]},"finish_reason":null}]}`,
+ `{"id":"x","model":"m","choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"function":{"arguments":"/x\"}"}}]},"finish_reason":null}]}`,
+ `{"id":"x","model":"m","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}`,
+ `[DONE]`,
+}
+
+// Interleaved calls reach a Responses client (Codex) as two function_call
+// items, one per index, in index order, each whole, with every event of
+// an item under the same output_index and item id.
+func TestInterleavedToolCallsToResponses(t *testing.T) {
+ out := chatToResponses(t, interleavedCalls...)
+ type item struct {
+ id, callID, name, args, streamed string
+ index float64
+ }
+ var items []*item
+ byID := map[string]*item{}
+ idAt := map[float64]string{}
+ for _, ev := range out {
+ switch ev["type"] {
+ case "response.output_item.added":
+ it, _ := ev["item"].(map[string]any)
+ if it["type"] != "function_call" {
+ continue
+ }
+ x := &item{id: it["id"].(string), index: ev["output_index"].(float64)}
+ items = append(items, x)
+ byID[x.id] = x
+ idAt[x.index] = x.id
+ case "response.function_call_arguments.delta":
+ x := byID[ev["item_id"].(string)]
+ if x == nil || idAt[ev["output_index"].(float64)] != x.id {
+ t.Fatalf("delta for an item not added at its output_index: %v", ev)
+ }
+ x.streamed += ev["delta"].(string)
+ case "response.output_item.done":
+ it, _ := ev["item"].(map[string]any)
+ if it["type"] != "function_call" {
+ continue
+ }
+ x := byID[it["id"].(string)]
+ if x == nil || idAt[ev["output_index"].(float64)] != x.id {
+ t.Fatalf("done for an item not added at its output_index: %v", ev)
+ }
+ x.callID, x.name, x.args = it["call_id"].(string), it["name"].(string), it["arguments"].(string)
+ }
+ }
+ if len(items) != 2 {
+ t.Fatalf("%d function_call items, want 2: %+v", len(items), out)
+ }
+ want := []struct{ callID, name, key, value string }{
+ {"call_read", "read_file", "path", "a/b.go"},
+ {"call_list", "list_dir", "dir", "src/x"},
+ }
+ for i, w := range want {
+ x := items[i]
+ var args map[string]string
+ if err := json.Unmarshal([]byte(x.args), &args); err != nil {
+ t.Fatalf("call %d arguments %q are not whole JSON: %v", i, x.args, err)
+ }
+ if x.callID != w.callID || x.name != w.name || args[w.key] != w.value || len(args) != 1 {
+ t.Fatalf("call %d = %s %s %s, want %s %s {%q:%q}", i, x.callID, x.name, x.args, w.callID, w.name, w.key, w.value)
+ }
+ if x.streamed != x.args {
+ t.Fatalf("call %d streamed %q but is done with %q", i, x.streamed, x.args)
+ }
+ }
+ if items[0].index == items[1].index || items[0].id == items[1].id {
+ t.Fatalf("two calls share an output_index or id: %+v %+v", items[0], items[1])
+ }
+}
+
+// The same stream seen by an Anthropic client: two tool_use blocks, each
+// with its own whole input.
+func TestInterleavedToolCallsToAnthropic(t *testing.T) {
+ out := chatToAnthropic(t, interleavedCalls...)
+ type block struct{ id, name, input string }
+ var blocks []*block
+ at := map[float64]*block{}
+ for _, ev := range out {
+ switch ev["type"] {
+ case "content_block_start":
+ if b, ok := ev["content_block"].(map[string]any); ok && b["type"] == "tool_use" {
+ x := &block{id: b["id"].(string), name: b["name"].(string)}
+ blocks = append(blocks, x)
+ at[ev["index"].(float64)] = x
+ }
+ case "content_block_delta":
+ if d, ok := ev["delta"].(map[string]any); ok && d["type"] == "input_json_delta" {
+ at[ev["index"].(float64)].input += d["partial_json"].(string)
+ }
+ }
+ }
+ if len(blocks) != 2 {
+ t.Fatalf("%d tool_use blocks, want 2: %+v", len(blocks), out)
+ }
+ got := blocks[0].id + " " + blocks[0].name + " " + blocks[0].input + " | " + blocks[1].id + " " + blocks[1].name + " " + blocks[1].input
+ if got != `call_read read_file {"path":"a/b.go"} | call_list list_dir {"dir":"src/x"}` {
+ t.Fatalf("blocks = %s", got)
+ }
+}
+
+// And a client answered whole, from the same stream: two calls.
+func TestInterleavedToolCallsWhole(t *testing.T) {
+ var d chatDecoder
+ var col collector
+ for _, c := range interleavedCalls {
+ if err := d.decode(c, col.add); err != nil {
+ t.Fatal(err)
+ }
+ }
+ var got []string
+ for _, p := range col.finish().Parts {
+ if p.Kind == ToolCall {
+ got = append(got, p.ID+" "+p.Name+" "+string(p.Args))
+ }
+ }
+ if strings.Join(got, " | ") != `call_read read_file {"path":"a/b.go"} | call_list list_dir {"dir":"src/x"}` {
+ t.Fatalf("calls = %q", got)
+ }
+}
+
+// A stream that names every call first, arguments empty, then gives each
+// call's arguments in turn, ending at [DONE] with no finish_reason: still
+// one call per index, each with its own arguments.
+func TestToolCallsNamedFirstThenArgued(t *testing.T) {
+ var d chatDecoder
+ var col collector
+ for _, c := range []string{
+ `{"id":"x","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"a","type":"function","function":{"name":"one","arguments":""}},{"index":1,"id":"b","type":"function","function":{"name":"two","arguments":""}}]}}]}`,
+ `{"id":"x","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"n\":1}"}}]}}]}`,
+ `{"id":"x","choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"function":{"arguments":"{\"n\":2}"}}]}}]}`,
+ `[DONE]`,
+ } {
+ if err := d.decode(c, col.add); err != nil {
+ t.Fatal(err)
+ }
+ }
+ var got []string
+ for _, p := range col.finish().Parts {
+ if p.Kind == ToolCall {
+ got = append(got, p.ID+" "+p.Name+" "+string(p.Args))
+ }
+ }
+ if strings.Join(got, " | ") != `a one {"n":1} | b two {"n":2}` {
+ t.Fatalf("calls = %q", got)
+ }
+}