Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

gateway: a Chat reply's cache read reaches the agent in OpenAI's names (B3TT3R on Discord)

yetonedefbb1a15996
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index b0d380547..afb48764e 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -127,6 +127,7 @@ recorded as a prediction failure (`noPrediction`).
- Each record keeps the service tier three ways (#1784): `tier_asked` is what the client's body asked (`bodyTier`; Claude's speed fast reads `fast`), `tier_sent` is what that try's last request went out with, noted in `forwardOnce` after every rewrite (fast picks, Ultrafast to priority on a plan without it) onto the per-attempt `wireAt`, and `tier_served` is only what the reply said it ran at (`upstreamTier`, read by the sniffer and the translated readers), empty when it says nothing. The route's tries carry `tier` and `tier_served` the same way. A bridge that doesn't go through `forwardOnce` (the Claude subscription, Cursor's built-in) leaves `tier_sent` empty. Code: [`tier_record.go`](../../internal/gateway/tier_record.go).
- An agent's fast pick (`settings.FastPicks`, the bolt in its picker, #954) may also be in `settings.UltraPicks` (#1954), offered by the GUI only where one of Codex's accounts lists Ultrafast on the model (`provider.CanUltra`, `UltraModels`). `fastPick` marks its candidates `ultra`; a try of one goes with `service_tier: "ultrafast"` on Chat and Responses, and an agent on Messages carries it in the context (`withUltra`) to `forwardTranslated`, which sets `Request.Ultrafast` for the ChatGPT backend. Either way the same check as a client's own Ultrafast ask applies, now on Chat as well as Responses: an account whose list doesn't offer it is put after a mate that does (`tierMate`), or, with none left, sent `priority`. A Chat body's own `service_tier: "ultrafast"` is parsed as Ultrafast too (`chat.go`). `TestUltraPickSentUltrafast`.
- A usage record carries its prompt's cache beside `in` and `out` (#1839): `cache_read` (tokens read from the vendor's cache) and `cache_write` (cache creation, Anthropic's `cache_creation_input_tokens`; `cache_write_1h` the part written for an hour). `in` holds neither. Both are written on every record, 0 too, so a prompt that missed the cache reads as that. A record from before then has neither and reads as 0. Every route fills them: Anthropic's messages, Chat (OpenAI's `prompt_tokens_details.cached_tokens`, DeepSeek's `prompt_cache_hit_tokens`), Responses (`input_tokens_details.cached_tokens`) and Gemini (`cachedContentTokenCount`), passed through or translated, whole or streamed, a plugin's fetch, and the Claude subscription bridge. System One's decision calls report no cache. `TestUsageRecordsCacheTokensOnEveryRoute`, `TestPluginUsageRecordsCacheTokens`, `TestClaudeSubscriptionUsageKeepsCacheTokens`, `TestRecordAlwaysCarriesCacheTokens`.
+- What the vendor read from its cache reaches the agent in the names its SDK reads, not only the record. A relayed Chat reply whose usage tells the cache in another vendor's names (a relay's Anthropic `cache_read_input_tokens` and `cache_creation_input_tokens`, DeepSeek's `prompt_cache_hit_tokens` or Moonshot's top-level `cached_tokens` beside a 0 in `prompt_tokens_details`) is told in OpenAI's too, streamed (`chatTidy`) and whole (`chatWhole`), by `openAIUsage` in [`chattidy.go`](../../internal/gateway/chattidy.go): `prompt_tokens_details.cached_tokens` (and `cache_write_tokens`), with `prompt_tokens` and `total_tokens` made to hold the cached part where the vendor left it out. Every other field stays, and a usage already in OpenAI's names, or with no cache, goes byte for byte; one in OpenAI's names alone isn't even decoded (`mayMendUsage`). pi-ai, which DeepSeek Harness and Pi use, reads `prompt_tokens_details.cached_tokens` first, a 0 there too, and maps no Anthropic names at all, so such a relay showed DSH 缓存命中 0% while magpie had counted the hit (B3TT3R on Discord asked; DeepSeek's own API sends both names and was already read right). Relayed Responses and Messages replies, translated replies and a plugin's need nothing: their usage is written in the agent's own names. `TestAgentReadsCacheTokensOnEveryRoute`, `TestPluginAgentReadsCacheTokens`, `TestOpenAIUsageLeavesOpenAIsAlone`, `TestMayMendUsageOnlyOtherNames` in [`usage_cache_agent_test.go`](../../internal/gateway/usage_cache_agent_test.go).
- A request whose key is a Claude sign-in (`claudeSignIn`: `Authorization: Bearer sk-ant-oat…`, Claude Code signed in to claude.ai with magpie as its base URL) is never told a 401, nor a 403 saying "OAuth token has been revoked". magpie never passes that sign-in on, so such an answer is a provider refusing magpie's credential, or one of magpie's own Claude accounts lapsing; told it, Claude Code renews its own sign-in before each of its retries (ten for one request) and then asks to `/login`. Any error whose message mentions x-api-key, whatever its status, Claude Code tells as "Not logged in · Please run /login" too. It is told as a 502 in magpie's words, an `api_error` saying "<provider> refused magpie's credential (HTTP 401): …" with x-api-key said as "API key", and any other error such a request is told keeps its status with x-api-key said as "API key", as does a stream's error after its 200 (`keepsSignIn`, `claude_signin.go`). Recent calls and the usage log keep the provider's status and message. Any other 403, and a request with magpie's key or an API key, is answered as before. The sign-in's beta (`oauth-…` in `anthropic-beta`) isn't passed on either (`askedBetas`), so a provider gets the request as Claude Code with magpie's key sends it; a provider's own `header.anthropic-beta` still goes.
- A 2xx that can't be an API's answer is the 502 it stands for (`notAnAPIReply`, #1012): a web page (`text/html`, or a body that begins as one: a sign-in page, Cloudflare's challenge), a body with nothing in it, or one sent as JSON that doesn't begin as JSON. Every upstream request goes through it (`forwardOnce`, and the built-in clients' own: the ChatGPT backend's `/responses`), so it fails over, rests and is logged as a 502 does, its reason naming the page's title. Only the first bytes are peeked, which a stream waits for anyway, and a compressed body is left as it came.
- A 2xx from Zhipu or Z.ai holding the platform's refusal envelope (`{"code":1000,"msg":…,"success":false}`, which their Responses API at `/api/v1` answers a refused key) is the refusal it stands for (`provider.ZhipuEnveloped` in `forwardOnce`): 401 for a key refused, 429 for a limit or no balance, as Zhipu's error codes give, so it fails over and rests as those do and the agent is told why. `TestZhipuResponsesRefusalIsRefusal`.
diff --git a/internal/gateway/chattidy.go b/internal/gateway/chattidy.go
index 24a7c5624..82ed7d0f5 100644
--- a/internal/gateway/chattidy.go
+++ b/internal/gateway/chattidy.go
@@ -61,6 +61,13 @@ import (
// never made (#3386). A fragment with none is given its call's: a new id,
// or a later place in its delta, is the next call.
//
+// And a usage that tells the cache only in another vendor's names
+// (Anthropic's cache_read_input_tokens from a relay in front of a model,
+// DeepSeek's prompt_cache_hit_tokens or Moonshot's cached_tokens beside a
+// 0 in prompt_tokens_details) is told in OpenAI's too (openAIUsage): the
+// gateway counted the hit, but an agent on pi-ai (DeepSeek Harness), which
+// reads prompt_tokens_details.cached_tokens first, showed none.
+//
// Lines pass whole, as they came, unless one has any of them.
type chatTidy struct {
buf []byte
@@ -128,9 +135,84 @@ func needsTidy(line []byte) bool {
bytes.Contains(line, toolCallsField)
}
-// needs is needsTidy, and a finish_reason once a choice made a call.
+// needs is needsTidy, a finish_reason once a choice made a call, and a
+// usage that may tell its cache in names OpenAI's clients don't read.
func (t *chatTidy) needs(b []byte) bool {
- return needsTidy(b) || len(t.called) > 0 && bytes.Contains(b, finishReasonField)
+ return needsTidy(b) || len(t.called) > 0 && bytes.Contains(b, finishReasonField) || mayMendUsage(b)
+}
+
+var (
+ usageField = []byte(`"usage"`)
+ cachedTokensField = []byte(`"cached_tokens"`)
+ // the names other than OpenAI's that cUsage reads a cache in
+ otherCacheFields = [][]byte{[]byte(`"prompt_cache_hit_tokens"`),
+ []byte(`"cache_read_input_tokens"`), []byte(`"cache_creation_input_tokens"`)}
+)
+
+// mayMendUsage reports whether b has a usage that may tell its cache in
+// names other than OpenAI's, which openAIUsage may have to tell again:
+// DeepSeek's or Anthropic's, or Moonshot's top-level cached_tokens beside
+// prompt_tokens_details' own. A usage in OpenAI's names alone, every
+// OpenAI reply, is left unread.
+func mayMendUsage(b []byte) bool {
+ if !bytes.Contains(b, usageField) {
+ return false
+ }
+ for _, f := range otherCacheFields {
+ if bytes.Contains(b, f) {
+ return true
+ }
+ }
+ return bytes.Count(b, cachedTokensField) > 1
+}
+
+// openAIUsage is a Chat usage whose cache cUsage reads in names other than
+// OpenAI's (Anthropic's from a relay, DeepSeek's prompt_cache_hit_tokens,
+// Moonshot's cached_tokens), told in OpenAI's as well: what was read in
+// prompt_tokens_details.cached_tokens, what was written in its
+// cache_write_tokens (OpenRouter's), and prompt_tokens counting both, as
+// OpenAI's does. An agent reads prompt_tokens_details first, pi-ai (DeepSeek
+// Harness) even a 0 there over DeepSeek's hit, and the AI SDK nothing else.
+// Every field the upstream sent stays; a usage that already says it in
+// OpenAI's names, or tells no cache, comes back as it was, and so does one
+// that isn't an object.
+func openAIUsage(raw json.RawMessage) (json.RawMessage, bool) {
+ var cu cUsage
+ if decodeJSON(raw, &cu) != nil {
+ return raw, false
+ }
+ u := cu.usage()
+ if u.CacheRead == 0 && u.CacheWrite == 0 {
+ return raw, false
+ }
+ in := u.prompt()
+ if d := cu.PromptTokensDetails; d != nil && d.CachedTokens == u.CacheRead && d.CacheWriteTokens == u.CacheWrite && cu.PromptTokens == in {
+ return raw, false
+ }
+ var m map[string]json.RawMessage
+ if decodeJSON(raw, &m) != nil || m == nil {
+ return raw, false
+ }
+ var details map[string]json.RawMessage
+ if decodeJSON(m["prompt_tokens_details"], &details) != nil || details == nil {
+ details = map[string]json.RawMessage{}
+ }
+ details["cached_tokens"], _ = marshalPlain(u.CacheRead)
+ if _, had := details["cache_write_tokens"]; had || u.CacheWrite > 0 {
+ details["cache_write_tokens"], _ = marshalPlain(u.CacheWrite)
+ }
+ m["prompt_tokens_details"], _ = marshalPlain(details)
+ if cu.PromptTokens != in {
+ m["prompt_tokens"], _ = marshalPlain(in)
+ if _, had := m["total_tokens"]; had {
+ m["total_tokens"], _ = marshalPlain(cu.TotalTokens + in - cu.PromptTokens)
+ }
+ }
+ out, err := marshalPlain(m)
+ if err != nil {
+ return raw, false
+ }
+ return out, true
}
var finishReasonField = []byte(`"finish_reason"`)
@@ -149,12 +231,19 @@ func (t *chatTidy) line(line []byte) []byte {
return line
}
var chunk map[string]json.RawMessage
- if decodeJSON(data, &chunk) != nil {
+ if decodeJSON(data, &chunk) != nil || chunk == nil {
return line
}
+ usageMended := false
+ if raw, ok := chunk["usage"]; ok {
+ chunk["usage"], usageMended = openAIUsage(raw)
+ }
var choices []map[string]json.RawMessage
if decodeJSON(chunk["choices"], &choices) != nil {
- return line
+ choices = nil
+ if !usageMended {
+ return line
+ }
}
changed := false
for k, c := range choices {
@@ -209,10 +298,12 @@ func (t *chatTidy) line(line []byte) []byte {
changed = true
}
}
- if !changed {
+ if !changed && !usageMended {
return line
}
- chunk["choices"], _ = json.Marshal(choices)
+ if changed {
+ chunk["choices"], _ = json.Marshal(choices)
+ }
nb, err := json.Marshal(chunk)
if err != nil {
return line
@@ -378,8 +469,9 @@ func flattenContent(m map[string]json.RawMessage, whole bool) bool {
// when the request offered tools (names), calls the model wrote into its
// text are made the calls they are, as textCallTidy makes them in a
// stream (#1845), and a call the upstream gave no id is given one, as
-// chatTidy gives it in a stream (#2611). It is held until it ends; any
-// other reply passes byte for byte.
+// chatTidy gives it in a stream (#2611), and a usage that tells its cache
+// in other names is told in OpenAI's too (openAIUsage). It is held until it
+// ends; any other reply passes byte for byte.
type chatWhole struct {
names map[string]bool
buf []byte
@@ -394,16 +486,23 @@ func (t *chatWhole) flush() []byte {
b := t.buf
t.buf = nil
parts := bytes.Contains(b, contentParts) || bytes.Contains(b, contentPartsSpace)
- if !parts && !mayWriteCalls(b, t.names) && !bytes.Contains(b, toolCallsField) {
+ if !parts && !mayWriteCalls(b, t.names) && !bytes.Contains(b, toolCallsField) && !mayMendUsage(b) {
return b
}
var reply map[string]json.RawMessage
- if decodeJSON(b, &reply) != nil {
+ if decodeJSON(b, &reply) != nil || reply == nil {
return b
}
+ usageMended := false
+ if raw, ok := reply["usage"]; ok {
+ reply["usage"], usageMended = openAIUsage(raw)
+ }
var choices []map[string]json.RawMessage
if decodeJSON(reply["choices"], &choices) != nil {
- return b
+ choices = nil
+ if !usageMended {
+ return b
+ }
}
changed := false
for _, c := range choices {
@@ -432,10 +531,12 @@ func (t *chatWhole) flush() []byte {
c["message"], _ = marshalPlain(msg)
changed = true
}
- if !changed {
+ if !changed && !usageMended {
return b
}
- reply["choices"], _ = marshalPlain(choices)
+ if changed {
+ reply["choices"], _ = marshalPlain(choices)
+ }
nb, err := marshalPlain(reply)
if err != nil {
return b
diff --git a/internal/gateway/usage_cache_agent_test.go b/internal/gateway/usage_cache_agent_test.go
new file mode 100644
index 000000000..fbc5b2d73
--- /dev/null
+++ b/internal/gateway/usage_cache_agent_test.go
@@ -0,0 +1,229 @@
+package gateway
+
+import (
+ "encoding/json"
+ "net/http/httptest"
+ "strings"
+ "testing"
+
+ "github.com/yetone/magpie/internal/provider"
+)
+
+// agentUsage is the prompt, cache read and cache write an agent reads from
+// a reply in the protocol it asked in, the way its SDK reads them: on Chat
+// as pi-ai does (DeepSeek Harness, Pi), prompt_tokens_details.cached_tokens
+// first, a 0 there too, then DeepSeek's prompt_cache_hit_tokens, then
+// cached_tokens; on Responses input_tokens_details; on Messages
+// Anthropic's own names. A stream's last usage that says any is what
+// counts.
+func agentUsage(path, body string) (prompt, read, write int) {
+ num := func(m map[string]any, k string) (int, bool) {
+ v, ok := m[k].(float64)
+ return int(v), ok
+ }
+ from := func(u map[string]any) (p, r, w int) {
+ switch {
+ case strings.HasSuffix(path, "/chat/completions"):
+ p, _ = num(u, "prompt_tokens")
+ d, _ := u["prompt_tokens_details"].(map[string]any)
+ var ok bool
+ if r, ok = num(d, "cached_tokens"); !ok {
+ if r, ok = num(u, "prompt_cache_hit_tokens"); !ok {
+ r, _ = num(u, "cached_tokens")
+ }
+ }
+ w, _ = num(d, "cache_write_tokens")
+ case strings.HasSuffix(path, "/responses"):
+ p, _ = num(u, "input_tokens")
+ d, _ := u["input_tokens_details"].(map[string]any)
+ r, _ = num(d, "cached_tokens")
+ w, _ = num(d, "cache_write_tokens")
+ default:
+ in, _ := num(u, "input_tokens")
+ r, _ = num(u, "cache_read_input_tokens")
+ w, _ = num(u, "cache_creation_input_tokens")
+ p = in + r + w
+ }
+ return
+ }
+ usages := func(j map[string]any) (out []map[string]any) {
+ if u, ok := j["usage"].(map[string]any); ok {
+ out = append(out, u)
+ }
+ for _, k := range []string{"response", "message"} {
+ if r, ok := j[k].(map[string]any); ok {
+ if u, ok := r["usage"].(map[string]any); ok {
+ out = append(out, u)
+ }
+ }
+ }
+ return out
+ }
+ docs := []string{body}
+ if strings.Contains(body, "data:") {
+ docs = nil
+ for _, l := range strings.Split(body, "\n") {
+ if d, ok := strings.CutPrefix(l, "data: "); ok {
+ docs = append(docs, d)
+ }
+ }
+ }
+ for _, d := range docs {
+ var j map[string]any
+ if json.Unmarshal([]byte(d), &j) != nil {
+ continue
+ }
+ for _, u := range usages(j) {
+ if p, r, w := from(u); r > read || p > prompt {
+ prompt, read, write = max(p, prompt), max(r, read), max(w, write)
+ }
+ }
+ }
+ return
+}
+
+// What a vendor read from its prompt cache reaches the agent in the words
+// its SDK reads, whatever names the vendor told it in, passed through or
+// translated, whole or streamed. A Chat relay that told it in Anthropic's
+// names, or DeepSeek's hits beside a 0 of OpenAI's, was passed through as
+// it came: the gateway counted the hit, and DeepSeek Harness, reading
+// prompt_tokens_details.cached_tokens, showed 缓存命中 0% (B3TT3R on
+// Discord).
+func TestAgentReadsCacheTokensOnEveryRoute(t *testing.T) {
+ vendors := []struct {
+ wire string
+ proto provider.Protocol
+ write int
+ }{
+ {"anthropic", provider.Anthropic, 40},
+ {"chat", provider.Chat, 0},
+ {"chat-anthropic-names", provider.Chat, 40},
+ {"chat-deepseek", provider.Chat, 0},
+ {"chat-zero-details", provider.Chat, 0},
+ {"chat-moonshot", provider.Chat, 0},
+ {"responses", provider.Responses, 0},
+ {"gemini", provider.Gemini, 0},
+ }
+ agents := []struct{ name, path, body string }{
+ {"messages", "/v1/messages", `{"model":"fake/m1","max_tokens":100,"messages":[{"role":"user","content":"hi"}]}`},
+ {"messages stream", "/v1/messages", `{"model":"fake/m1","stream":true,"max_tokens":100,"messages":[{"role":"user","content":"hi"}]}`},
+ {"chat", "/v1/chat/completions", `{"model":"fake/m1","messages":[{"role":"user","content":"hi"}]}`},
+ {"chat stream", "/v1/chat/completions", `{"model":"fake/m1","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"hi"}]}`},
+ {"responses", "/v1/responses", `{"model":"fake/m1","input":"hi"}`},
+ {"responses stream", "/v1/responses", `{"model":"fake/m1","stream":true,"input":"hi"}`},
+ }
+ for _, v := range vendors {
+ for _, a := range agents {
+ t.Run(v.wire+" to "+a.name, func(t *testing.T) {
+ fresh(t)
+ up := httptest.NewServer(cachingVendor{v.wire})
+ t.Cleanup(up.Close)
+ p := provider.Provider{ID: "fake", Name: "Fake", Key: "k", Models: []string{"m1"}}
+ switch v.proto {
+ case provider.Chat:
+ p.Chat = up.URL + "/v1"
+ case provider.Responses:
+ p.Responses = up.URL + "/v1"
+ case provider.Anthropic:
+ p.Anthropic = up.URL
+ case provider.Gemini:
+ p.Gemini = up.URL + "/v1beta"
+ }
+ if err := provider.Save(p); err != nil {
+ t.Fatal(err)
+ }
+ rec := httptest.NewRecorder()
+ New().Handler().ServeHTTP(rec, httptest.NewRequest("POST", a.path, strings.NewReader(a.body)))
+ if rec.Code != 200 || !strings.Contains(rec.Body.String(), "hi") {
+ t.Fatalf("%d %s", rec.Code, rec.Body)
+ }
+ // the whole prompt: what was read, written and neither
+ const wantPrompt = 352
+ prompt, read, write := agentUsage(a.path, rec.Body.String())
+ if read != 300 || write != v.write || prompt != wantPrompt {
+ t.Errorf("agent reads prompt %d, cache read %d, cache write %d; want %d, 300, %d\n%s",
+ prompt, read, write, wantPrompt, v.write, rec.Body)
+ }
+ })
+ }
+ }
+}
+
+// A usage told in OpenAI's names already passes byte for byte, and so does
+// one with no cache: openAIUsage mends only what an agent would misread.
+func TestOpenAIUsageLeavesOpenAIsAlone(t *testing.T) {
+ for _, u := range []string{
+ `{"prompt_tokens":352,"completion_tokens":7,"total_tokens":359,"prompt_tokens_details":{"cached_tokens":300}}`,
+ `{"prompt_tokens":52,"completion_tokens":7,"total_tokens":59}`,
+ `{"prompt_tokens":52,"completion_tokens":7,"total_tokens":59,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":0}`,
+ `null`,
+ `"x"`,
+ } {
+ if out, changed := openAIUsage(json.RawMessage(u)); changed || string(out) != u {
+ t.Errorf("%s came back %s (changed %v)", u, out, changed)
+ }
+ }
+ // a stream's chunk with the usage alone keeps every field it had
+ line := []byte(`data: {"id":"c1","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":7,"total_tokens":19,"cache_read_input_tokens":300,"cache_creation_input_tokens":40}}` + "\n")
+ got := string((&chatTidy{}).write(line))
+ for _, want := range []string{`"prompt_tokens":352`, `"total_tokens":359`, `"cached_tokens":300`, `"cache_write_tokens":40`,
+ `"cache_read_input_tokens":300`, `"cache_creation_input_tokens":40`, `"choices":[]`, `"id":"c1"`} {
+ if !strings.Contains(got, want) {
+ t.Errorf("mended chunk lacks %s: %s", want, got)
+ }
+ }
+}
+
+// A usage told only in OpenAI's names is never read again: every OpenAI
+// reply carries one, and mayMendUsage keeps the decode to the replies
+// that may need it. Moonshot's top-level cached_tokens beside a 0 of
+// prompt_tokens_details' is told in OpenAI's too.
+func TestMayMendUsageOnlyOtherNames(t *testing.T) {
+ for b, want := range map[string]bool{
+ `{"usage":{"prompt_tokens":352,"prompt_tokens_details":{"cached_tokens":300}}}`: false,
+ `{"usage":{"prompt_tokens":352}}`: false,
+ `{"usage":{"prompt_tokens":352,"prompt_cache_hit_tokens":300}}`: true,
+ `{"usage":{"prompt_tokens":12,"cache_read_input_tokens":300}}`: true,
+ `{"usage":{"prompt_tokens":352,"prompt_tokens_details":{"cached_tokens":0},"cached_tokens":300}}`: true,
+ `{"choices":[],"prompt_cache_hit_tokens":300}`: false,
+ } {
+ if got := mayMendUsage([]byte(b)); got != want {
+ t.Errorf("mayMendUsage(%s) = %v, want %v", b, got, want)
+ }
+ }
+ out, changed := openAIUsage(json.RawMessage(`{"prompt_tokens":352,"completion_tokens":7,"total_tokens":359,"prompt_tokens_details":{"cached_tokens":0},"cached_tokens":300}`))
+ if !changed || !strings.Contains(string(out), `"prompt_tokens_details":{"cached_tokens":300}`) || !strings.Contains(string(out), `"prompt_tokens":352`) {
+ t.Errorf("Moonshot's usage came back %s (changed %v)", out, changed)
+ }
+}
+
+// A plugin's reply reaches a Chat agent with its cache in OpenAI's names
+// too, whatever its model speaks: the agent reads what the vendor read.
+func TestPluginAgentReadsCacheTokens(t *testing.T) {
+ movedFake(t, "factory", cachingVendor{})
+ for _, c := range []struct {
+ model string
+ write int
+ }{
+ {"fake-1", 0}, // @ai-sdk/openai-compatible: Chat
+ {"fake-claude", 40}, // @ai-sdk/anthropic: Messages
+ {"fake-gemini", 0}, // @ai-sdk/google: generateContent
+ } {
+ for _, stream := range []bool{false, true} {
+ body := `{"model":"factory/` + c.model + `","messages":[{"role":"user","content":"hi"}]}`
+ if stream {
+ body = `{"model":"factory/` + c.model + `","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"hi"}]}`
+ }
+ rec := httptest.NewRecorder()
+ New().Handler().ServeHTTP(rec, httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(body)))
+ if rec.Code != 200 || !strings.Contains(rec.Body.String(), "hi") {
+ t.Fatalf("%s stream=%v: %d %s", c.model, stream, rec.Code, rec.Body)
+ }
+ prompt, read, write := agentUsage("/v1/chat/completions", rec.Body.String())
+ if read != 300 || write != c.write || prompt != 352 {
+ t.Errorf("%s stream=%v: agent reads prompt %d, cache read %d, cache write %d; want 352, 300, %d\n%s",
+ c.model, stream, prompt, read, write, c.write, rec.Body)
+ }
+ }
+ }
+}
diff --git a/internal/gateway/usage_cache_test.go b/internal/gateway/usage_cache_test.go
index 7ebeda054..dc22bf985 100644
--- a/internal/gateway/usage_cache_test.go
+++ b/internal/gateway/usage_cache_test.go
@@ -49,7 +49,7 @@ func (v cachingVendor) ServeHTTP(w http.ResponseWriter, r *http.Request) {
"event: message_delta\ndata: " + `{"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":7}}`,
"event: message_stop\ndata: " + `{"type":"message_stop"}`,
}
- case "chat", "chat-anthropic-names", "chat-deepseek":
+ case "chat", "chat-anthropic-names", "chat-deepseek", "chat-zero-details", "chat-moonshot":
u := map[string]string{
// OpenAI's: prompt_tokens counts the cached ones too
"chat": `{"prompt_tokens":352,"completion_tokens":7,"total_tokens":359,"prompt_tokens_details":{"cached_tokens":300}}`,
@@ -57,6 +57,10 @@ func (v cachingVendor) ServeHTTP(w http.ResponseWriter, r *http.Request) {
"chat-anthropic-names": `{"prompt_tokens":12,"completion_tokens":7,"total_tokens":19,"cache_read_input_tokens":300,"cache_creation_input_tokens":40}`,
// DeepSeek's hits and misses
"chat-deepseek": `{"prompt_tokens":352,"completion_tokens":7,"total_tokens":359,"prompt_cache_hit_tokens":300,"prompt_cache_miss_tokens":52}`,
+ // a relay that puts DeepSeek's hits beside a 0 of OpenAI's
+ "chat-zero-details": `{"prompt_tokens":352,"completion_tokens":7,"total_tokens":359,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":300,"prompt_cache_miss_tokens":52}`,
+ // Moonshot's top-level cached_tokens beside a 0 of OpenAI's
+ "chat-moonshot": `{"prompt_tokens":352,"completion_tokens":7,"total_tokens":359,"prompt_tokens_details":{"cached_tokens":0},"cached_tokens":300}`,
}[v.wire]
whole = `{"id":"c1","object":"chat.completion","model":"m1","choices":[{"index":0,"message":{"role":"assistant","content":"hi"},"finish_reason":"stop"}],"usage":` + u + `}`
events = []string{
@@ -111,6 +115,8 @@ func TestUsageRecordsCacheTokensOnEveryRoute(t *testing.T) {
{"chat", provider.Chat, want{52, 300, 0}},
{"chat-anthropic-names", provider.Chat, want{12, 300, 40}},
{"chat-deepseek", provider.Chat, want{52, 300, 0}},
+ {"chat-zero-details", provider.Chat, want{52, 300, 0}},
+ {"chat-moonshot", provider.Chat, want{52, 300, 0}},
{"responses", provider.Responses, want{52, 300, 0}},
{"gemini", provider.Gemini, want{52, 300, 0}},
}