Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
gateway: a Chat reply's cache read reaches the agent in OpenAI's names (B3TT3R on Discord)
yetonedefbb1a15996
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex b0d380547..afb48764e 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -127,6 +127,7 @@ recorded as a prediction failure (`noPrediction`).- Each record keeps the service tier three ways (#1784): `tier_asked` is what the client's body asked (`bodyTier`; Claude's speed fast reads `fast`), `tier_sent` is what that try's last request went out with, noted in `forwardOnce` after every rewrite (fast picks, Ultrafast to priority on a plan without it) onto the per-attempt `wireAt`, and `tier_served` is only what the reply said it ran at (`upstreamTier`, read by the sniffer and the translated readers), empty when it says nothing. The route's tries carry `tier` and `tier_served` the same way. A bridge that doesn't go through `forwardOnce` (the Claude subscription, Cursor's built-in) leaves `tier_sent` empty. Code: [`tier_record.go`](../../internal/gateway/tier_record.go).- An agent's fast pick (`settings.FastPicks`, the bolt in its picker, #954) may also be in `settings.UltraPicks` (#1954), offered by the GUI only where one of Codex's accounts lists Ultrafast on the model (`provider.CanUltra`, `UltraModels`). `fastPick` marks its candidates `ultra`; a try of one goes with `service_tier: "ultrafast"` on Chat and Responses, and an agent on Messages carries it in the context (`withUltra`) to `forwardTranslated`, which sets `Request.Ultrafast` for the ChatGPT backend. Either way the same check as a client's own Ultrafast ask applies, now on Chat as well as Responses: an account whose list doesn't offer it is put after a mate that does (`tierMate`), or, with none left, sent `priority`. A Chat body's own `service_tier: "ultrafast"` is parsed as Ultrafast too (`chat.go`). `TestUltraPickSentUltrafast`.- A usage record carries its prompt's cache beside `in` and `out` (#1839): `cache_read` (tokens read from the vendor's cache) and `cache_write` (cache creation, Anthropic's `cache_creation_input_tokens`; `cache_write_1h` the part written for an hour). `in` holds neither. Both are written on every record, 0 too, so a prompt that missed the cache reads as that. A record from before then has neither and reads as 0. Every route fills them: Anthropic's messages, Chat (OpenAI's `prompt_tokens_details.cached_tokens`, DeepSeek's `prompt_cache_hit_tokens`), Responses (`input_tokens_details.cached_tokens`) and Gemini (`cachedContentTokenCount`), passed through or translated, whole or streamed, a plugin's fetch, and the Claude subscription bridge. System One's decision calls report no cache. `TestUsageRecordsCacheTokensOnEveryRoute`, `TestPluginUsageRecordsCacheTokens`, `TestClaudeSubscriptionUsageKeepsCacheTokens`, `TestRecordAlwaysCarriesCacheTokens`.+- What the vendor read from its cache reaches the agent in the names its SDK reads, not only the record. A relayed Chat reply whose usage tells the cache in another vendor's names (a relay's Anthropic `cache_read_input_tokens` and `cache_creation_input_tokens`, DeepSeek's `prompt_cache_hit_tokens` or Moonshot's top-level `cached_tokens` beside a 0 in `prompt_tokens_details`) is told in OpenAI's too, streamed (`chatTidy`) and whole (`chatWhole`), by `openAIUsage` in [`chattidy.go`](../../internal/gateway/chattidy.go): `prompt_tokens_details.cached_tokens` (and `cache_write_tokens`), with `prompt_tokens` and `total_tokens` made to hold the cached part where the vendor left it out. Every other field stays, and a usage already in OpenAI's names, or with no cache, goes byte for byte; one in OpenAI's names alone isn't even decoded (`mayMendUsage`). pi-ai, which DeepSeek Harness and Pi use, reads `prompt_tokens_details.cached_tokens` first, a 0 there too, and maps no Anthropic names at all, so such a relay showed DSH 缓存命中 0% while magpie had counted the hit (B3TT3R on Discord asked; DeepSeek's own API sends both names and was already read right). Relayed Responses and Messages replies, translated replies and a plugin's need nothing: their usage is written in the agent's own names. `TestAgentReadsCacheTokensOnEveryRoute`, `TestPluginAgentReadsCacheTokens`, `TestOpenAIUsageLeavesOpenAIsAlone`, `TestMayMendUsageOnlyOtherNames` in [`usage_cache_agent_test.go`](../../internal/gateway/usage_cache_agent_test.go).- A request whose key is a Claude sign-in (`claudeSignIn`: `Authorization: Bearer sk-ant-oat…`, Claude Code signed in to claude.ai with magpie as its base URL) is never told a 401, nor a 403 saying "OAuth token has been revoked". magpie never passes that sign-in on, so such an answer is a provider refusing magpie's credential, or one of magpie's own Claude accounts lapsing; told it, Claude Code renews its own sign-in before each of its retries (ten for one request) and then asks to `/login`. Any error whose message mentions x-api-key, whatever its status, Claude Code tells as "Not logged in · Please run /login" too. It is told as a 502 in magpie's words, an `api_error` saying "<provider> refused magpie's credential (HTTP 401): …" with x-api-key said as "API key", and any other error such a request is told keeps its status with x-api-key said as "API key", as does a stream's error after its 200 (`keepsSignIn`, `claude_signin.go`). Recent calls and the usage log keep the provider's status and message. Any other 403, and a request with magpie's key or an API key, is answered as before. The sign-in's beta (`oauth-…` in `anthropic-beta`) isn't passed on either (`askedBetas`), so a provider gets the request as Claude Code with magpie's key sends it; a provider's own `header.anthropic-beta` still goes.- A 2xx that can't be an API's answer is the 502 it stands for (`notAnAPIReply`, #1012): a web page (`text/html`, or a body that begins as one: a sign-in page, Cloudflare's challenge), a body with nothing in it, or one sent as JSON that doesn't begin as JSON. Every upstream request goes through it (`forwardOnce`, and the built-in clients' own: the ChatGPT backend's `/responses`), so it fails over, rests and is logged as a 502 does, its reason naming the page's title. Only the first bytes are peeked, which a stream waits for anyway, and a compressed body is left as it came.- A 2xx from Zhipu or Z.ai holding the platform's refusal envelope (`{"code":1000,"msg":…,"success":false}`, which their Responses API at `/api/v1` answers a refused key) is the refusal it stands for (`provider.ZhipuEnveloped` in `forwardOnce`): 401 for a key refused, 429 for a limit or no balance, as Zhipu's error codes give, so it fails over and rests as those do and the agent is told why. `TestZhipuResponsesRefusalIsRefusal`.diff --git a/internal/gateway/chattidy.go b/internal/gateway/chattidy.goindex 24a7c5624..82ed7d0f5 100644--- a/internal/gateway/chattidy.go+++ b/internal/gateway/chattidy.go@@ -61,6 +61,13 @@ import (// never made (#3386). A fragment with none is given its call's: a new id,// or a later place in its delta, is the next call.//+// And a usage that tells the cache only in another vendor's names+// (Anthropic's cache_read_input_tokens from a relay in front of a model,+// DeepSeek's prompt_cache_hit_tokens or Moonshot's cached_tokens beside a+// 0 in prompt_tokens_details) is told in OpenAI's too (openAIUsage): the+// gateway counted the hit, but an agent on pi-ai (DeepSeek Harness), which+// reads prompt_tokens_details.cached_tokens first, showed none.+//// Lines pass whole, as they came, unless one has any of them.type chatTidy struct {buf []byte@@ -128,9 +135,84 @@ func needsTidy(line []byte) bool {bytes.Contains(line, toolCallsField)}-// needs is needsTidy, and a finish_reason once a choice made a call.+// needs is needsTidy, a finish_reason once a choice made a call, and a+// usage that may tell its cache in names OpenAI's clients don't read.func (t *chatTidy) needs(b []byte) bool {- return needsTidy(b) || len(t.called) > 0 && bytes.Contains(b, finishReasonField)+ return needsTidy(b) || len(t.called) > 0 && bytes.Contains(b, finishReasonField) || mayMendUsage(b)+}++var (+ usageField = []byte(`"usage"`)+ cachedTokensField = []byte(`"cached_tokens"`)+ // the names other than OpenAI's that cUsage reads a cache in+ otherCacheFields = [][]byte{[]byte(`"prompt_cache_hit_tokens"`),+ []byte(`"cache_read_input_tokens"`), []byte(`"cache_creation_input_tokens"`)}+)++// mayMendUsage reports whether b has a usage that may tell its cache in+// names other than OpenAI's, which openAIUsage may have to tell again:+// DeepSeek's or Anthropic's, or Moonshot's top-level cached_tokens beside+// prompt_tokens_details' own. A usage in OpenAI's names alone, every+// OpenAI reply, is left unread.+func mayMendUsage(b []byte) bool {+ if !bytes.Contains(b, usageField) {+ return false+ }+ for _, f := range otherCacheFields {+ if bytes.Contains(b, f) {+ return true+ }+ }+ return bytes.Count(b, cachedTokensField) > 1+}++// openAIUsage is a Chat usage whose cache cUsage reads in names other than+// OpenAI's (Anthropic's from a relay, DeepSeek's prompt_cache_hit_tokens,+// Moonshot's cached_tokens), told in OpenAI's as well: what was read in+// prompt_tokens_details.cached_tokens, what was written in its+// cache_write_tokens (OpenRouter's), and prompt_tokens counting both, as+// OpenAI's does. An agent reads prompt_tokens_details first, pi-ai (DeepSeek+// Harness) even a 0 there over DeepSeek's hit, and the AI SDK nothing else.+// Every field the upstream sent stays; a usage that already says it in+// OpenAI's names, or tells no cache, comes back as it was, and so does one+// that isn't an object.+func openAIUsage(raw json.RawMessage) (json.RawMessage, bool) {+ var cu cUsage+ if decodeJSON(raw, &cu) != nil {+ return raw, false+ }+ u := cu.usage()+ if u.CacheRead == 0 && u.CacheWrite == 0 {+ return raw, false+ }+ in := u.prompt()+ if d := cu.PromptTokensDetails; d != nil && d.CachedTokens == u.CacheRead && d.CacheWriteTokens == u.CacheWrite && cu.PromptTokens == in {+ return raw, false+ }+ var m map[string]json.RawMessage+ if decodeJSON(raw, &m) != nil || m == nil {+ return raw, false+ }+ var details map[string]json.RawMessage+ if decodeJSON(m["prompt_tokens_details"], &details) != nil || details == nil {+ details = map[string]json.RawMessage{}+ }+ details["cached_tokens"], _ = marshalPlain(u.CacheRead)+ if _, had := details["cache_write_tokens"]; had || u.CacheWrite > 0 {+ details["cache_write_tokens"], _ = marshalPlain(u.CacheWrite)+ }+ m["prompt_tokens_details"], _ = marshalPlain(details)+ if cu.PromptTokens != in {+ m["prompt_tokens"], _ = marshalPlain(in)+ if _, had := m["total_tokens"]; had {+ m["total_tokens"], _ = marshalPlain(cu.TotalTokens + in - cu.PromptTokens)+ }+ }+ out, err := marshalPlain(m)+ if err != nil {+ return raw, false+ }+ return out, true}var finishReasonField = []byte(`"finish_reason"`)@@ -149,12 +231,19 @@ func (t *chatTidy) line(line []byte) []byte {return line}var chunk map[string]json.RawMessage- if decodeJSON(data, &chunk) != nil {+ if decodeJSON(data, &chunk) != nil || chunk == nil {return line}+ usageMended := false+ if raw, ok := chunk["usage"]; ok {+ chunk["usage"], usageMended = openAIUsage(raw)+ }var choices []map[string]json.RawMessageif decodeJSON(chunk["choices"], &choices) != nil {- return line+ choices = nil+ if !usageMended {+ return line+ }}changed := falsefor k, c := range choices {@@ -209,10 +298,12 @@ func (t *chatTidy) line(line []byte) []byte {changed = true}}- if !changed {+ if !changed && !usageMended {return line}- chunk["choices"], _ = json.Marshal(choices)+ if changed {+ chunk["choices"], _ = json.Marshal(choices)+ }nb, err := json.Marshal(chunk)if err != nil {return line@@ -378,8 +469,9 @@ func flattenContent(m map[string]json.RawMessage, whole bool) bool {// when the request offered tools (names), calls the model wrote into its// text are made the calls they are, as textCallTidy makes them in a// stream (#1845), and a call the upstream gave no id is given one, as-// chatTidy gives it in a stream (#2611). It is held until it ends; any-// other reply passes byte for byte.+// chatTidy gives it in a stream (#2611), and a usage that tells its cache+// in other names is told in OpenAI's too (openAIUsage). It is held until it+// ends; any other reply passes byte for byte.type chatWhole struct {names map[string]boolbuf []byte@@ -394,16 +486,23 @@ func (t *chatWhole) flush() []byte {b := t.buft.buf = nilparts := bytes.Contains(b, contentParts) || bytes.Contains(b, contentPartsSpace)- if !parts && !mayWriteCalls(b, t.names) && !bytes.Contains(b, toolCallsField) {+ if !parts && !mayWriteCalls(b, t.names) && !bytes.Contains(b, toolCallsField) && !mayMendUsage(b) {return b}var reply map[string]json.RawMessage- if decodeJSON(b, &reply) != nil {+ if decodeJSON(b, &reply) != nil || reply == nil {return b}+ usageMended := false+ if raw, ok := reply["usage"]; ok {+ reply["usage"], usageMended = openAIUsage(raw)+ }var choices []map[string]json.RawMessageif decodeJSON(reply["choices"], &choices) != nil {- return b+ choices = nil+ if !usageMended {+ return b+ }}changed := falsefor _, c := range choices {@@ -432,10 +531,12 @@ func (t *chatWhole) flush() []byte {c["message"], _ = marshalPlain(msg)changed = true}- if !changed {+ if !changed && !usageMended {return b}- reply["choices"], _ = marshalPlain(choices)+ if changed {+ reply["choices"], _ = marshalPlain(choices)+ }nb, err := marshalPlain(reply)if err != nil {return bdiff --git a/internal/gateway/usage_cache_agent_test.go b/internal/gateway/usage_cache_agent_test.gonew file mode 100644index 000000000..fbc5b2d73--- /dev/null+++ b/internal/gateway/usage_cache_agent_test.go@@ -0,0 +1,229 @@+package gateway++import (+ "encoding/json"+ "net/http/httptest"+ "strings"+ "testing"++ "github.com/yetone/magpie/internal/provider"+)++// agentUsage is the prompt, cache read and cache write an agent reads from+// a reply in the protocol it asked in, the way its SDK reads them: on Chat+// as pi-ai does (DeepSeek Harness, Pi), prompt_tokens_details.cached_tokens+// first, a 0 there too, then DeepSeek's prompt_cache_hit_tokens, then+// cached_tokens; on Responses input_tokens_details; on Messages+// Anthropic's own names. A stream's last usage that says any is what+// counts.+func agentUsage(path, body string) (prompt, read, write int) {+ num := func(m map[string]any, k string) (int, bool) {+ v, ok := m[k].(float64)+ return int(v), ok+ }+ from := func(u map[string]any) (p, r, w int) {+ switch {+ case strings.HasSuffix(path, "/chat/completions"):+ p, _ = num(u, "prompt_tokens")+ d, _ := u["prompt_tokens_details"].(map[string]any)+ var ok bool+ if r, ok = num(d, "cached_tokens"); !ok {+ if r, ok = num(u, "prompt_cache_hit_tokens"); !ok {+ r, _ = num(u, "cached_tokens")+ }+ }+ w, _ = num(d, "cache_write_tokens")+ case strings.HasSuffix(path, "/responses"):+ p, _ = num(u, "input_tokens")+ d, _ := u["input_tokens_details"].(map[string]any)+ r, _ = num(d, "cached_tokens")+ w, _ = num(d, "cache_write_tokens")+ default:+ in, _ := num(u, "input_tokens")+ r, _ = num(u, "cache_read_input_tokens")+ w, _ = num(u, "cache_creation_input_tokens")+ p = in + r + w+ }+ return+ }+ usages := func(j map[string]any) (out []map[string]any) {+ if u, ok := j["usage"].(map[string]any); ok {+ out = append(out, u)+ }+ for _, k := range []string{"response", "message"} {+ if r, ok := j[k].(map[string]any); ok {+ if u, ok := r["usage"].(map[string]any); ok {+ out = append(out, u)+ }+ }+ }+ return out+ }+ docs := []string{body}+ if strings.Contains(body, "data:") {+ docs = nil+ for _, l := range strings.Split(body, "\n") {+ if d, ok := strings.CutPrefix(l, "data: "); ok {+ docs = append(docs, d)+ }+ }+ }+ for _, d := range docs {+ var j map[string]any+ if json.Unmarshal([]byte(d), &j) != nil {+ continue+ }+ for _, u := range usages(j) {+ if p, r, w := from(u); r > read || p > prompt {+ prompt, read, write = max(p, prompt), max(r, read), max(w, write)+ }+ }+ }+ return+}++// What a vendor read from its prompt cache reaches the agent in the words+// its SDK reads, whatever names the vendor told it in, passed through or+// translated, whole or streamed. A Chat relay that told it in Anthropic's+// names, or DeepSeek's hits beside a 0 of OpenAI's, was passed through as+// it came: the gateway counted the hit, and DeepSeek Harness, reading+// prompt_tokens_details.cached_tokens, showed 缓存命中 0% (B3TT3R on+// Discord).+func TestAgentReadsCacheTokensOnEveryRoute(t *testing.T) {+ vendors := []struct {+ wire string+ proto provider.Protocol+ write int+ }{+ {"anthropic", provider.Anthropic, 40},+ {"chat", provider.Chat, 0},+ {"chat-anthropic-names", provider.Chat, 40},+ {"chat-deepseek", provider.Chat, 0},+ {"chat-zero-details", provider.Chat, 0},+ {"chat-moonshot", provider.Chat, 0},+ {"responses", provider.Responses, 0},+ {"gemini", provider.Gemini, 0},+ }+ agents := []struct{ name, path, body string }{+ {"messages", "/v1/messages", `{"model":"fake/m1","max_tokens":100,"messages":[{"role":"user","content":"hi"}]}`},+ {"messages stream", "/v1/messages", `{"model":"fake/m1","stream":true,"max_tokens":100,"messages":[{"role":"user","content":"hi"}]}`},+ {"chat", "/v1/chat/completions", `{"model":"fake/m1","messages":[{"role":"user","content":"hi"}]}`},+ {"chat stream", "/v1/chat/completions", `{"model":"fake/m1","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"hi"}]}`},+ {"responses", "/v1/responses", `{"model":"fake/m1","input":"hi"}`},+ {"responses stream", "/v1/responses", `{"model":"fake/m1","stream":true,"input":"hi"}`},+ }+ for _, v := range vendors {+ for _, a := range agents {+ t.Run(v.wire+" to "+a.name, func(t *testing.T) {+ fresh(t)+ up := httptest.NewServer(cachingVendor{v.wire})+ t.Cleanup(up.Close)+ p := provider.Provider{ID: "fake", Name: "Fake", Key: "k", Models: []string{"m1"}}+ switch v.proto {+ case provider.Chat:+ p.Chat = up.URL + "/v1"+ case provider.Responses:+ p.Responses = up.URL + "/v1"+ case provider.Anthropic:+ p.Anthropic = up.URL+ case provider.Gemini:+ p.Gemini = up.URL + "/v1beta"+ }+ if err := provider.Save(p); err != nil {+ t.Fatal(err)+ }+ rec := httptest.NewRecorder()+ New().Handler().ServeHTTP(rec, httptest.NewRequest("POST", a.path, strings.NewReader(a.body)))+ if rec.Code != 200 || !strings.Contains(rec.Body.String(), "hi") {+ t.Fatalf("%d %s", rec.Code, rec.Body)+ }+ // the whole prompt: what was read, written and neither+ const wantPrompt = 352+ prompt, read, write := agentUsage(a.path, rec.Body.String())+ if read != 300 || write != v.write || prompt != wantPrompt {+ t.Errorf("agent reads prompt %d, cache read %d, cache write %d; want %d, 300, %d\n%s",+ prompt, read, write, wantPrompt, v.write, rec.Body)+ }+ })+ }+ }+}++// A usage told in OpenAI's names already passes byte for byte, and so does+// one with no cache: openAIUsage mends only what an agent would misread.+func TestOpenAIUsageLeavesOpenAIsAlone(t *testing.T) {+ for _, u := range []string{+ `{"prompt_tokens":352,"completion_tokens":7,"total_tokens":359,"prompt_tokens_details":{"cached_tokens":300}}`,+ `{"prompt_tokens":52,"completion_tokens":7,"total_tokens":59}`,+ `{"prompt_tokens":52,"completion_tokens":7,"total_tokens":59,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":0}`,+ `null`,+ `"x"`,+ } {+ if out, changed := openAIUsage(json.RawMessage(u)); changed || string(out) != u {+ t.Errorf("%s came back %s (changed %v)", u, out, changed)+ }+ }+ // a stream's chunk with the usage alone keeps every field it had+ line := []byte(`data: {"id":"c1","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":7,"total_tokens":19,"cache_read_input_tokens":300,"cache_creation_input_tokens":40}}` + "\n")+ got := string((&chatTidy{}).write(line))+ for _, want := range []string{`"prompt_tokens":352`, `"total_tokens":359`, `"cached_tokens":300`, `"cache_write_tokens":40`,+ `"cache_read_input_tokens":300`, `"cache_creation_input_tokens":40`, `"choices":[]`, `"id":"c1"`} {+ if !strings.Contains(got, want) {+ t.Errorf("mended chunk lacks %s: %s", want, got)+ }+ }+}++// A usage told only in OpenAI's names is never read again: every OpenAI+// reply carries one, and mayMendUsage keeps the decode to the replies+// that may need it. Moonshot's top-level cached_tokens beside a 0 of+// prompt_tokens_details' is told in OpenAI's too.+func TestMayMendUsageOnlyOtherNames(t *testing.T) {+ for b, want := range map[string]bool{+ `{"usage":{"prompt_tokens":352,"prompt_tokens_details":{"cached_tokens":300}}}`: false,+ `{"usage":{"prompt_tokens":352}}`: false,+ `{"usage":{"prompt_tokens":352,"prompt_cache_hit_tokens":300}}`: true,+ `{"usage":{"prompt_tokens":12,"cache_read_input_tokens":300}}`: true,+ `{"usage":{"prompt_tokens":352,"prompt_tokens_details":{"cached_tokens":0},"cached_tokens":300}}`: true,+ `{"choices":[],"prompt_cache_hit_tokens":300}`: false,+ } {+ if got := mayMendUsage([]byte(b)); got != want {+ t.Errorf("mayMendUsage(%s) = %v, want %v", b, got, want)+ }+ }+ out, changed := openAIUsage(json.RawMessage(`{"prompt_tokens":352,"completion_tokens":7,"total_tokens":359,"prompt_tokens_details":{"cached_tokens":0},"cached_tokens":300}`))+ if !changed || !strings.Contains(string(out), `"prompt_tokens_details":{"cached_tokens":300}`) || !strings.Contains(string(out), `"prompt_tokens":352`) {+ t.Errorf("Moonshot's usage came back %s (changed %v)", out, changed)+ }+}++// A plugin's reply reaches a Chat agent with its cache in OpenAI's names+// too, whatever its model speaks: the agent reads what the vendor read.+func TestPluginAgentReadsCacheTokens(t *testing.T) {+ movedFake(t, "factory", cachingVendor{})+ for _, c := range []struct {+ model string+ write int+ }{+ {"fake-1", 0}, // @ai-sdk/openai-compatible: Chat+ {"fake-claude", 40}, // @ai-sdk/anthropic: Messages+ {"fake-gemini", 0}, // @ai-sdk/google: generateContent+ } {+ for _, stream := range []bool{false, true} {+ body := `{"model":"factory/` + c.model + `","messages":[{"role":"user","content":"hi"}]}`+ if stream {+ body = `{"model":"factory/` + c.model + `","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"hi"}]}`+ }+ rec := httptest.NewRecorder()+ New().Handler().ServeHTTP(rec, httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(body)))+ if rec.Code != 200 || !strings.Contains(rec.Body.String(), "hi") {+ t.Fatalf("%s stream=%v: %d %s", c.model, stream, rec.Code, rec.Body)+ }+ prompt, read, write := agentUsage("/v1/chat/completions", rec.Body.String())+ if read != 300 || write != c.write || prompt != 352 {+ t.Errorf("%s stream=%v: agent reads prompt %d, cache read %d, cache write %d; want 352, 300, %d\n%s",+ c.model, stream, prompt, read, write, c.write, rec.Body)+ }+ }+ }+}diff --git a/internal/gateway/usage_cache_test.go b/internal/gateway/usage_cache_test.goindex 7ebeda054..dc22bf985 100644--- a/internal/gateway/usage_cache_test.go+++ b/internal/gateway/usage_cache_test.go@@ -49,7 +49,7 @@ func (v cachingVendor) ServeHTTP(w http.ResponseWriter, r *http.Request) {"event: message_delta\ndata: " + `{"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":7}}`,"event: message_stop\ndata: " + `{"type":"message_stop"}`,}- case "chat", "chat-anthropic-names", "chat-deepseek":+ case "chat", "chat-anthropic-names", "chat-deepseek", "chat-zero-details", "chat-moonshot":u := map[string]string{// OpenAI's: prompt_tokens counts the cached ones too"chat": `{"prompt_tokens":352,"completion_tokens":7,"total_tokens":359,"prompt_tokens_details":{"cached_tokens":300}}`,@@ -57,6 +57,10 @@ func (v cachingVendor) ServeHTTP(w http.ResponseWriter, r *http.Request) {"chat-anthropic-names": `{"prompt_tokens":12,"completion_tokens":7,"total_tokens":19,"cache_read_input_tokens":300,"cache_creation_input_tokens":40}`,// DeepSeek's hits and misses"chat-deepseek": `{"prompt_tokens":352,"completion_tokens":7,"total_tokens":359,"prompt_cache_hit_tokens":300,"prompt_cache_miss_tokens":52}`,+ // a relay that puts DeepSeek's hits beside a 0 of OpenAI's+ "chat-zero-details": `{"prompt_tokens":352,"completion_tokens":7,"total_tokens":359,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":300,"prompt_cache_miss_tokens":52}`,+ // Moonshot's top-level cached_tokens beside a 0 of OpenAI's+ "chat-moonshot": `{"prompt_tokens":352,"completion_tokens":7,"total_tokens":359,"prompt_tokens_details":{"cached_tokens":0},"cached_tokens":300}`,}[v.wire]whole = `{"id":"c1","object":"chat.completion","model":"m1","choices":[{"index":0,"message":{"role":"assistant","content":"hi"},"finish_reason":"stop"}],"usage":` + u + `}`events = []string{@@ -111,6 +115,8 @@ func TestUsageRecordsCacheTokensOnEveryRoute(t *testing.T) {{"chat", provider.Chat, want{52, 300, 0}},{"chat-anthropic-names", provider.Chat, want{12, 300, 40}},{"chat-deepseek", provider.Chat, want{52, 300, 0}},+ {"chat-zero-details", provider.Chat, want{52, 300, 0}},+ {"chat-moonshot", provider.Chat, want{52, 300, 0}},{"responses", provider.Responses, want{52, 300, 0}},{"gemini", provider.Gemini, want{52, 300, 0}},}