Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
gateway: a Responses request may name the response it follows (previous_response_id, kevinzhow on X)
yetoneba3a7384fb2a
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex 3b4f01339..c5d19e682 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -1157,6 +1157,46 @@ the row of the entry that id reaches, with its label, under the bare id.`internal/gui/tests/bare-ids.test.cjs`.- A URL magpie downloads but didn't choose itself is downloaded from public addresses only, wherever the request came from ([`image_fetch_guard.go`](../../internal/gateway/image_fetch_guard.go)). That is an image a caller names (an `/v1/images/edits` or `/v1/images/generations` reference image, for an images API that takes files, Gemini's own API or Code Assist: `fetchPicture` in [`draw.go`](../../internal/gateway/draw.go), `publicClient`), a page `web_fetch` reads (`webFetchClient`), and an image or video a vendor answers by URL, which what the model was asked to draw or do can steer (the inline download for a `b64_json` caller in `draw.go` and `/v1/videos/{id}/content` in [`video.go`](../../internal/gateway/video.go), `vendorClient`; generate_image's own download in [`internal/imagemcp`](../../internal/imagemcp/imagemcp.go), `PublicClient`). Every address the host resolves to must be public (`publicAddr`, which is [`internal/netguard`](../../internal/netguard/netguard.go)'s `Public`, the rule a provider's icon uses too: not loopback, RFC 1918, fc00::/7, link-local with 169.254.169.254, CGNAT 100.64/10, unspecified, multicast, documentation, reserved, benchmarking, Teredo, nor an IPv4-mapped, NAT64 or 6to4 form of one; an AAAA record Loon synthesizes for an A answer in the same reply, `fd27:712::/96` with that IPv4 address inside, is left out and the A answer judged in its place, `dropLoon`), and the request goes to the address checked, so a name that resolves elsewhere the second time changes nothing; a direct connection is checked again on the address dialled (`guardDialer`). Through a proxy the proxy is sent that address, never the name (an `https://` CONNECT or SOCKS keeps the name for TLS and Host; a plain `http://` URL through an HTTP proxy is asked for by the address in Host too). A fake-IP resolver's answer is the one shape let past that (#2593; ggbdpq, with h4rk8s's and TomXPRIME's reports): Clash's TUN with fake-ip, Clash Verge, Surge's enhanced mode and Loon answer every name in 198.18.0.0/15 (`netguard.FakeIP`), so with one on every download was refused as a private address. A name (never an address literal) whose every answer is in that range is fetched: with no proxy (pure TUN) its fake address is pinned and dialled, the dial guard letting that one address in, and the TUN carries it; through a proxy on this computer (a loopback address or `localhost`, `onThisComputer`) the proxy is given the name, never the fake address, which a real proxy can't route (h4rk8s: a CONNECT by the name went through where one by its 198.18 address failed), and a plain `http://` request's Host is the checked name, never the caller's own; through a proxy elsewhere it is refused. An answer with any other address beside the fake ones (private or public) is judged as any, so a fake one beside 192.168.x is refused. The trust this asks for is the fake-IP resolver's: which host the name stands for, and so where the TUN or the proxy connects, is decided where magpie can't see, and a name that really points at the user's network is reached if that resolver's rules send it there. A provider's icon was already fetched this way (`publicIP` in [`internal/provider/icon.go`](../../internal/provider/icon.go), #252), and shares the rule. `TestFakeIPNameGoesToALocalProxyByName`, `TestFakeIPNameDialledThroughTheTUN`, `TestFakeIPRefusesTheRest` in [`fakeip_test.go`](../../internal/gateway/fakeip_test.go). Each redirect, up to 5, is checked the same, and a caller's image must sniff as an image. A request from this computer gets nothing more: a reverse proxy or relay in front of magpie on the same host makes every request it carries arrive from loopback, so where a request came from is never a reason to reach a private address. Two things are: settings.json's `fetchPrivate` (`settings.FetchPrivate`, none by default, no Settings page, this computer's own and never brought by a sync or restore), a list of names, `name:port`s, IPs and CIDR ranges the user lets magpie download from (`allowFor`; a settings.json that can't be read lets nothing extra in), and for a vendor's URL the `host:port` of that provider's own endpoints, which the user named themselves (a provider on the LAN that serves its images from itself). A vendor's image that can't be downloaded is answered by its URL, never dropped; its video is a 502 naming why. A URL left for the vendor to fetch (a chat model's message) is the vendor's to fetch.+## A Responses request that names the response it follows++A Responses client may send only its new input with `previous_response_id`+(kevinzhow on X: every turn sent the whole conversation again, images+included). `previousResponse` in+[`prev_response.go`](../../internal/gateway/prev_response.go) runs in+`serveAgent` before anything else reads the body.++- Every Responses reply magpie passes with status 200, not failed or+ cancelled, is kept by its id with the conversation it answered (input,+ then output) in `prevResponses`: in memory only, least recently used+ first out past 256 MB, each kept 6 hours (`prevTTL`, `prevCap`). A+ request with `store: false` keeps nothing. Codex over HTTP sends+ `store: false` with the whole input and never names a previous response+ (only its websocket request has the field), so nothing changes for it.+- A turn is kept for the caller that made it (`prevScope`): its gateway+ key, the remote magpie it came through, and the address of a keyless+ remote caller. Another caller naming the id is told it isn't there.+- A request naming a kept id is given the conversation whole+ (`previous_response_id` taken out, the kept items before its own input),+ so every route, translation and masking below sees what it would have+ seen had the client sent it all. The stored items are what the client+ sent, and masking runs on each request as it goes, so a stored secret+ is masked the same way on every later turn.+- An id magpie doesn't keep (never seen, another caller's, expired, or+ lost to a restart) gets OpenAI's 400 `previous_response_not_found`+ (`writePrevNotFound`) on a route that translates, before anything is+ sent. A Responses relay is sent the request as it came, and its own+ answer reaches the client.+- Where the provider keeps responses itself (`prevKeeps`: an API key, no+ plugin, on api.openai.com, Azure, or Ark's `/api/v3` pay-as-you-go+ Responses) and it answered the previous turn on the same key+ (`prevRoute`), with `store` not off, the request goes as the client's+ new input with the provider's own id (`prevDelta`), so images aren't+ uploaded again. A provider that says it doesn't know that id+ (`prevRefused`) is sent the whole conversation at once.+- Not covered: the kept turns don't outlive the gateway, and a kept-id+ request is still read and masked whole on this computer before the+ delta is cut.+## Verification```sh@@ -1164,6 +1204,7 @@ go test -tags nogui ./internal/gateway -run 'TestGroupAsksOneRestingMemberNotEacgo test -tags nogui ./internal/gateway -run 'TestMemberTooSmallIsNotAsked|TestMemberTooSmallIsTheLastResort|TestClaudeAccountsSpentFallBackToCodex|TestConversationOutgrowingItsMemberMoves|TestRoomyLeavesWhatItCantJudge|TestPayloadTooLarge|TestOverflow|TestContextOverflow|TestRelayInputCap|TestEveryVendorsOverflow'go test -tags nogui ./internal/gateway -run 'TestPlain403|TestOrderedKeepsAnAccountAt98|TestPromptRefusalRestsNobody'go test -tags nogui ./internal/gateway -run 'TestVertex'+go test -tags nogui ./internal/gateway -run 'TestPreviousResponse|TestUnknownPreviousResponse|TestPrevStore'go test -tags nogui ./internal/gateway -run 'TestBedrock'go test -tags nogui ./internal/gateway -run 'TestPublicAddr|TestEditImageURL|TestPrivateDialer|TestFetchPrivateEntries|TestVendorImageURLAtPrivateAddress|TestVideoContentAtPrivateAddress'go test -tags nogui ./internal/gateway ./internal/provider ./internal/cli ./internal/gui -run 'TestDisabled|TestModelDisableCmd'diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.goindex 5fea4c7ac..410ad2292 100644--- a/internal/gateway/gateway.go+++ b/internal/gateway/gateway.go@@ -4050,6 +4050,13 @@ func (s *Server) attempt(w http.ResponseWriter, r *http.Request, from provider.P// translation and a plugin's fetch alikebody = chatCallIDs(body)}+ // a response the request named that magpie doesn't hold goes only to+ // a provider it is relayed to as written (prev_response.go): Claude's+ // and the ChatGPT backend's accounts drop it, as a translation does+ noteAnswered(r, "")+ if id := prevLost(r); id != "" && p.Account != nil && (p.Account.Agent == "claude" || p.Account.Agent == "codex") {+ return writePrevNotFound(w, id)+ }// every request to the provider goes through its own proxy, if it has// one (#237)r = r.WithContext(p.Via(r.Context()))@@ -4190,6 +4197,9 @@ func (s *Server) attempt(w http.ResponseWriter, r *http.Request, from provider.Pif ownAPI && slices.Contains(to, from) && !firstElsewhere(p, model, from) {to = []provider.Protocol{from}}+ if id := prevLost(r); id != "" {+ return writePrevNotFound(w, id)+ }call.To = to[0]return s.translate(w, r, p, from, to[0], model, body, &call.Usage)}@@ -4865,10 +4875,31 @@ func (s *Server) passthrough(w http.ResponseWriter, r *http.Request, p provider.if proto == provider.Responses && p.Account != nil && (p.Account.Agent == "grok" || p.PluginProvider() == "grok" || p.PluginProvider() == "zed") {named = namespacedIn(body)}+ // the new input alone, after the id the provider keeps the previous+ // response under, when it answered that one (prev_response.go)+ sentWhole, delta := body, false+ if proto == provider.Responses {+ body, delta = prevDelta(r, p, body)+ }res, err := s.forward(r.Context(), p, proto, path, p.Prepare(body), r.Header)if err != nil {return writeError(w, proto, 502, p.Name+": "+err.Error()), err.Error(), true}+ if delta && badRequest(res.StatusCode) || delta && res.StatusCode == http.StatusNotFound {+ // one it no longer holds: the conversation whole+ b, _ := io.ReadAll(io.LimitReader(res.Body, 1<<20))+ res.Body.Close()+ res.Body = io.NopCloser(bytes.NewReader(b))+ if prevRefused(res.StatusCode, b) {+ body = sentWhole+ if res, err = s.forward(r.Context(), p, proto, path, p.Prepare(body), r.Header); err != nil {+ return writeError(w, proto, 502, p.Name+": "+err.Error()), err.Error(), true+ }+ }+ }+ if proto == provider.Responses && prevKeptBy(p) && !storeOff(body) {+ noteAnswered(r, prevRoute(p))+ }// structured output turned away (Xiaomi MiMo's Token Plan takes no// json_schema, only text and json_object, #2126): a schema is asked// for as any JSON object with the schema in the system prompt, and ifdiff --git a/internal/gateway/image_body_bench_test.go b/internal/gateway/image_body_bench_test.gonew file mode 100644index 000000000..b4699dea4--- /dev/null+++ b/internal/gateway/image_body_bench_test.go@@ -0,0 +1,194 @@+package gateway++import (+ "bytes"+ "context"+ "encoding/base64"+ "encoding/json"+ "fmt"+ "image"+ "image/color"+ "image/png"+ "io"+ "math/rand/v2"+ "net/http"+ "net/http/httptest"+ "sync/atomic"+ "testing"++ "github.com/yetone/magpie/internal/provider"+)++// noisePNG is a real PNG of w×h random pixels: it doesn't compress, so its+// base64 is as large as a dense screenshot's of the same size.+func noisePNG(w, h int, seed uint64) []byte {+ r := rand.New(rand.NewPCG(seed, seed))+ img := image.NewNRGBA(image.Rect(0, 0, w, h))+ for i := 0; i < len(img.Pix); i += 4 {+ v := r.Uint32()+ img.Pix[i], img.Pix[i+1], img.Pix[i+2], img.Pix[i+3] = uint8(v), uint8(v>>8), uint8(v>>16), 255+ }+ _ = color.White+ var b bytes.Buffer+ png.Encode(&b, img)+ return b.Bytes()+}++// imageConversation is a conversation of turns turns, each with one image+// of about 2.6MB base64, as its client sends it.+func imageConversation(proto provider.Protocol, model string, turns int) []byte {+ type m = map[string]any+ var msgs []any+ for i := range turns {+ url := "data:image/png;base64," + base64.StdEncoding.EncodeToString(noisePNG(900, 700, uint64(i+1)))+ text := "what does this screenshot show?"+ switch proto {+ case provider.Anthropic:+ data := url[len("data:image/png;base64,"):]+ msgs = append(msgs, m{"role": "user", "content": []any{m{"type": "image", "source": m{"type": "base64", "media_type": "image/png", "data": data}}, m{"type": "text", "text": text}}},+ m{"role": "assistant", "content": []any{m{"type": "text", "text": "It shows noise."}}})+ case provider.Chat:+ msgs = append(msgs, m{"role": "user", "content": []any{m{"type": "image_url", "image_url": m{"url": url}}, m{"type": "text", "text": text}}},+ m{"role": "assistant", "content": "It shows noise."})+ default:+ msgs = append(msgs, m{"type": "message", "role": "user", "content": []any{m{"type": "input_image", "image_url": url}, m{"type": "input_text", "text": text}}},+ m{"type": "message", "role": "assistant", "content": []any{m{"type": "output_text", "text": "It shows noise."}}})+ }+ }+ body := m{"model": model, "stream": false}+ switch proto {+ case provider.Anthropic:+ msgs = append(msgs, m{"role": "user", "content": "go on"})+ body["max_tokens"] = 1000+ body["messages"] = msgs+ case provider.Chat:+ msgs = append(msgs, m{"role": "user", "content": "go on"})+ body["messages"] = msgs+ default:+ msgs = append(msgs, m{"type": "message", "role": "user", "content": []any{m{"type": "input_text", "text": "go on"}}})+ body["input"] = msgs+ }+ b, _ := json.Marshal(body)+ return b+}++func BenchmarkImageRequest(b *testing.B) {+ chatReply := `{"id":"c1","object":"chat.completion","choices":[{"index":0,"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}],"usage":{"prompt_tokens":7,"completion_tokens":1}}`+ chatSSE := "data: {\"id\":\"c1\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"ok\"},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":7,\"completion_tokens\":1}}\n\ndata: [DONE]\n\n"+ for _, tc := range []struct {+ name, path string+ in, up provider.Protocol+ reply string+ sse bool+ }{+ {"responses-to-chat", "/v1/responses", provider.Responses, provider.Chat, chatSSE, true},+ {"chat-to-chat", "/v1/chat/completions", provider.Chat, provider.Chat, chatReply, false},+ {"messages-to-chat", "/v1/messages", provider.Anthropic, provider.Chat, chatSSE, true},+ {"responses-to-responses", "/v1/responses", provider.Responses, provider.Responses, `{"id":"r1","object":"response","status":"completed","output":[{"type":"message","role":"assistant","content":[{"type":"output_text","text":"ok"}]}],"usage":{"input_tokens":7,"output_tokens":1}}`, false},+ } {+ b.Run(tc.name, func(b *testing.B) {+ b.Setenv("XDG_CONFIG_HOME", b.TempDir())+ b.Setenv("XDG_CACHE_HOME", b.TempDir())+ f := &fake{ctype: "application/json", reply: tc.reply}+ if tc.sse {+ f.ctype = ""+ }+ up := httptest.NewServer(f)+ defer up.Close()+ p := provider.Provider{ID: "fake", Name: "Fake", Key: "k", Models: []string{"m1"}}+ switch tc.up {+ case provider.Chat:+ p.Chat = up.URL + "/v1"+ case provider.Responses:+ p.Responses = up.URL + "/v1"+ }+ if err := provider.Save(p); err != nil {+ b.Fatal(err)+ }+ body := imageConversation(tc.in, "fake/m1", 8)+ s := New()+ b.SetBytes(int64(len(body)))+ b.ResetTimer()+ for range b.N {+ ctx, done := context.WithCancel(context.Background())+ rec := httptest.NewRecorder()+ s.Handler().ServeHTTP(rec, httptest.NewRequest("POST", tc.path, bytes.NewReader(body)).WithContext(ctx))+ done()+ if rec.Code != 200 || f.calls == 0 {+ b.Fatalf("%d %s", rec.Code, rec.Body.String())+ }+ }+ b.ReportMetric(float64(len(body))/1e6, "MB/req")+ })+ }+}++// BenchmarkPreviousResponseTurn is a turn after an 8-image conversation+// that a client sends as previous_response_id and its new text alone+// (kevinzhow on X), against the whole conversation sent again (whole):+// to a provider that keeps responses, which is sent the new input after+// its own id (kept), and to one on Chat, which magpie gives the whole+// conversation (chat). up-MB/req is what went upstream.+func BenchmarkPreviousResponseTurn(b *testing.B) {+ for _, tc := range []struct {+ name string+ chat, whole bool+ }{{"whole-to-responses", false, true}, {"prev-to-kept-responses", false, false}, {"prev-to-chat", true, false}} {+ b.Run(tc.name, func(b *testing.B) {+ b.Setenv("XDG_CONFIG_HOME", b.TempDir())+ b.Setenv("XDG_CACHE_HOME", b.TempDir())+ defer func(old func(provider.Provider) bool) { prevKeptBy = old }(prevKeptBy)+ prevKeptBy = func(p provider.Provider) bool { return p.ID == "fake" }+ var n, upBytes atomic.Int64+ up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ got, _ := io.ReadAll(r.Body)+ upBytes.Add(int64(len(got)))+ i := n.Add(1)+ if tc.chat {+ w.Header().Set("Content-Type", "text/event-stream")+ io.WriteString(w, "data: {\"id\":\"c1\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"ok\"},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n")+ return+ }+ w.Header().Set("Content-Type", "application/json")+ fmt.Fprintf(w, `{"id":"resp_%d","object":"response","status":"completed","output":[{"type":"message","role":"assistant","content":[{"type":"output_text","text":"ok"}]}],"usage":{"input_tokens":7,"output_tokens":1}}`, i)+ }))+ defer up.Close()+ p := provider.Provider{ID: "fake", Name: "Fake", Key: "k", Models: []string{"m1"}}+ if tc.chat {+ p.Chat = up.URL + "/v1"+ } else {+ p.Responses = up.URL + "/v1"+ }+ if err := provider.Save(p); err != nil {+ b.Fatal(err)+ }+ s := New()+ ask := func(body []byte) *httptest.ResponseRecorder {+ rec := httptest.NewRecorder()+ req := httptest.NewRequest("POST", "/v1/responses", bytes.NewReader(body))+ req.RemoteAddr = "127.0.0.1:5555"+ s.Handler().ServeHTTP(rec, req)+ if rec.Code != 200 {+ b.Fatalf("%d %.300s", rec.Code, rec.Body.String())+ }+ return rec+ }+ whole := imageConversation(provider.Responses, "fake/m1", 8)+ var seed struct{ ID string }+ json.Unmarshal(ask(whole).Body.Bytes(), &seed)+ turn := []byte(`{"model":"fake/m1","stream":false,"previous_response_id":"` + seed.ID + `","input":"go on"}`)+ if tc.whole {+ turn = whole+ }+ upBytes.Store(0)+ b.SetBytes(int64(len(turn)))+ b.ResetTimer()+ for range b.N {+ ask(turn)+ }+ b.StopTimer()+ b.ReportMetric(float64(len(turn))/1e6, "client-MB/req")+ b.ReportMetric(float64(upBytes.Load())/float64(b.N)/1e6, "up-MB/req")+ })+ }+}diff --git a/internal/gateway/middleware.go b/internal/gateway/middleware.goindex c46ac21f6..dc99ad997 100644--- a/internal/gateway/middleware.go+++ b/internal/gateway/middleware.go@@ -21,6 +21,10 @@ import (// magpie's to read.func (s *Server) serveAgent(w http.ResponseWriter, r *http.Request, from provider.Protocol, body []byte) {withGatewaySession(w, r)+ // a Responses request naming the response it follows is given the+ // conversation whole before anything reads it (prev_response.go)+ w, r, body, kept := previousResponse(w, r, from, body)+ defer kept()w, recorded := recordConversation(w, r, from, body)defer recorded()// a conversation going round one tool call is answered here, beforediff --git a/internal/gateway/prev_response.go b/internal/gateway/prev_response.gonew file mode 100644index 000000000..4a6cc655a--- /dev/null+++ b/internal/gateway/prev_response.go@@ -0,0 +1,431 @@+package gateway++import (+ "bytes"+ "container/list"+ "context"+ "crypto/sha256"+ "encoding/hex"+ "encoding/json"+ "net"+ "net/http"+ "strings"+ "sync"+ "time"++ "github.com/tidwall/gjson"+ "github.com/yetone/magpie/internal/access"+ "github.com/yetone/magpie/internal/provider"+)++// A Responses client may send only a turn's new input and name the+// response it follows (previous_response_id), as OpenAI's own API lets it+// (kevinzhow on X: every request carried the whole conversation, images+// and all). magpie keeps each response it gave such a client, its input+// and its output, under the response's id, and a request that names one+// is given the conversation whole before anything else reads it: a+// translation, a hook, the masking. A provider that keeps responses itself+// (prevKeeps), and answered the one named, is sent the new input alone+// after its own id, so what it holds isn't sent again.+//+// What is kept is the client's own form, before masking: the conversation+// it is given back into goes through the masking as every request does.+// It is kept in memory alone, bounded (prevCap) and for prevTTL, under the+// caller's gateway key (prevScope): one key never reads another's.+// A client that says store false (Codex over HTTP, whose requests never+// name a previous response: codex-rs/codex-api/src/common.rs keeps+// previous_response_id to its websocket's response.create) has nothing+// kept.++const (+ prevTTL = 6 * time.Hour+ // prevCap bounds the bytes kept, counted as each turn adds them: a+ // turn's history is the earlier turn's items, shared, not copied+ prevCap = 256 << 20+ // prevReplyMax is the most of one reply read for its output+ prevReplyMax = 64 << 20+)++type prevKey struct{ scope, id string }++// prevTurn is one response kept: the conversation up to and with it.+type prevTurn struct {+ key prevKey+ items []json.RawMessage // input, then the response's output+ own int // bytes this turn added to what it follows+ at time.Time+ // the provider that keeps it too, if one does: the id it gave it+ // and which provider and key (prevRoute)+ upstreamID, route string+}++type prevStore struct {+ mu sync.Mutex+ m map[prevKey]*list.Element+ order list.List // front the newest+ size int+ cap int+ ttl time.Duration+ now func() time.Time+}++var prevResponses = newPrevStore(prevCap, prevTTL)++func newPrevStore(cap int, ttl time.Duration) *prevStore {+ return &prevStore{m: map[prevKey]*list.Element{}, cap: cap, ttl: ttl, now: time.Now}+}++func (s *prevStore) get(k prevKey) (*prevTurn, bool) {+ s.mu.Lock()+ defer s.mu.Unlock()+ e, ok := s.m[k]+ if !ok {+ return nil, false+ }+ t := e.Value.(*prevTurn)+ if s.now().Sub(t.at) > s.ttl {+ s.drop(e)+ return nil, false+ }+ s.order.MoveToFront(e)+ return t, true+}++func (s *prevStore) put(t *prevTurn) {+ if t.own > s.cap/2 {+ return // one turn would push out nearly everything+ }+ s.mu.Lock()+ defer s.mu.Unlock()+ if e, ok := s.m[t.key]; ok {+ s.drop(e)+ }+ t.at = s.now()+ s.m[t.key] = s.order.PushFront(t)+ s.size += t.own+ for s.size > s.cap {+ s.drop(s.order.Back())+ }+ // and what has outlived its time, from the oldest+ for e := s.order.Back(); e != nil && s.now().Sub(e.Value.(*prevTurn).at) > s.ttl; e = s.order.Back() {+ s.drop(e)+ }+}++func (s *prevStore) drop(e *list.Element) {+ t := e.Value.(*prevTurn)+ s.order.Remove(e)+ delete(s.m, t.key)+ s.size -= t.own+}++// prevScope is whose responses a request may name: its gateway key, the+// computer a remote magpie passed it on from, and for a request with no+// key from another computer, that computer's address.+func prevScope(r *http.Request) string {+ keyID := access.Caller(r.Context()).KeyID+ s := "key:" + keyID+ if via := callerOf(r).via; via != "" {+ s += "\x00via:" + via+ }+ if keyID == "" && !local(r) {+ host, _, err := net.SplitHostPort(r.RemoteAddr)+ if err != nil {+ host = r.RemoteAddr+ }+ s += "\x00ip:" + host+ }+ return s+}++// prevRoute is the provider and API key a response came from: one the+// provider keeps is named to it again only with the same key.+func prevRoute(p provider.Provider) string {+ sum := sha256.Sum256([]byte(p.Key))+ return p.ID + "\x00" + hex.EncodeToString(sum[:8])+}++// prevKeeps is whether p keeps the responses it gives and takes+// previous_response_id: OpenAI's API and Azure OpenAI, and Ark's own+// /api/v3 (tried with a real key on 10-29: a stored response's word came+// back after its id, and an unknown id is a 400 PreviousResponseNotFound).+// A subscription's or a plugin's account is never asked so.+var prevKeptBy = prevKeeps++func prevKeeps(p provider.Provider) bool {+ if p.Account != nil || p.PluginProvider() != "" {+ return false+ }+ host := p.Host()+ if host == "api.openai.com" || p.IsAzure() {+ return true+ }+ return host == "ark.cn-beijing.volces.com" && strings.Contains(p.Base(provider.Responses), "/api/v3")+}++// prevAsk is what serveAgent made of a request's previous_response_id,+// for the tries of it (attempt, passthrough).+type prevAsk struct {+ id string // as the client named it+ known bool // the conversation was given back whole+ // the items of the whole conversation, and how many at its end are+ // the client's own new input+ total, fresh int+ upstreamID string // the id the provider that answered it keeps it under+ route string+ // which try answered: its route if it was relayed to a provider+ // that keeps what it is sent, empty otherwise+ mu sync.Mutex+ answered string+}++type prevAskKey struct{}++func prevAskOf(ctx context.Context) *prevAsk {+ a, _ := ctx.Value(prevAskKey{}).(*prevAsk)+ return a+}++// prevLost is the id a request named that magpie doesn't hold, which a+// try that doesn't relay it to a provider as written can't honour.+func prevLost(r *http.Request) string {+ if a := prevAskOf(r.Context()); a != nil && a.id != "" && !a.known {+ return a.id+ }+ return ""+}++// writePrevNotFound is OpenAI's own answer to an id it doesn't hold.+func writePrevNotFound(w http.ResponseWriter, id string) (int, string) {+ msg := "Previous response with id '" + id + "' not found."+ return writeErrorSaid(w, provider.Responses, 400, msg, "previous_response_not_found", "previous_response_id"), msg+}++// noteAnswered is called by each try: the last is the one that answered.+func noteAnswered(r *http.Request, route string) {+ if a := prevAskOf(r.Context()); a != nil {+ a.mu.Lock()+ a.answered = route+ a.mu.Unlock()+ }+}++// previousResponse gives a Responses request that names a response magpie+// holds the conversation whole, and keeps the reply for the next turn.+// Its done is deferred by the caller.+func previousResponse(w http.ResponseWriter, r *http.Request, from provider.Protocol, body []byte) (http.ResponseWriter, *http.Request, []byte, func()) {+ nothing := func() {}+ if from != provider.Responses {+ return w, r, body, nothing+ }+ var q map[string]json.RawMessage+ if decodeJSONView(body, &q) != nil {+ return w, r, body, nothing+ }+ scope := prevScope(r)+ ask := &prevAsk{}+ var prior []json.RawMessage+ if raw := q["previous_response_id"]; len(raw) > 0 && !bytes.Equal(raw, []byte("null")) {+ var id string+ if json.Unmarshal(raw, &id) == nil && id != "" {+ ask.id = id+ if t, ok := prevResponses.get(prevKey{scope, id}); ok {+ prior = t.items+ ask.known, ask.upstreamID, ask.route = true, t.upstreamID, t.route+ }+ }+ }+ input, fresh := prevInput(q["input"])+ if fresh == nil {+ // an input magpie can't read as items: left as it came+ return w, r, body, nothing+ }+ if ask.known {+ input = append(prior[:len(prior):len(prior)], fresh...)+ if b, err := json.Marshal(input); err == nil {+ q["input"] = b+ delete(q, "previous_response_id")+ if nb, err := json.Marshal(q); err == nil {+ body = nb+ }+ }+ ask.total, ask.fresh = len(input), len(fresh)+ }+ if ask.id != "" {+ r = r.WithContext(context.WithValue(r.Context(), prevAskKey{}, ask))+ }+ var store bool+ if json.Unmarshal(q["store"], &store) == nil && !store {+ return w, r, body, nothing // the client asked that nothing be kept+ }+ // what this turn adds: its whole body, which its items hold+ own := len(body)+ if ask.known {+ own = 0+ for _, it := range fresh {+ own += len(it)+ }+ }+ if ask.id == "" {+ // the answer note is wanted for a request naming none too+ r = r.WithContext(context.WithValue(r.Context(), prevAskKey{}, ask))+ }+ sink := &conversationWriter{captureResponseWriter: &captureResponseWriter{+ ResponseWriter: w, full: &spool{limit: prevReplyMax, pattern: "magpie-prevresp-*"},+ }}+ return sink, r, body, func() {+ defer sink.full.discard()+ if sink.status != 0 && sink.status != http.StatusOK || sink.full.cut() {+ return+ }+ reply, err := sink.full.read()+ if err != nil || len(reply) == 0 {+ return+ }+ id, output, ok := prevReply(sink.Header().Get("Content-Type"), reply)+ if !ok {+ return+ }+ for _, it := range output {+ own += len(it)+ }+ t := &prevTurn{key: prevKey{scope, id}, items: append(input[:len(input):len(input)], output...), own: own}+ ask.mu.Lock()+ if ask.answered != "" {+ t.upstreamID, t.route = id, ask.answered+ }+ ask.mu.Unlock()+ prevResponses.put(t)+ }+}++// prevInput is a Responses input as items: a string is one user message.+// fresh is nil for an input that is neither. The items share the body's+// bytes, which nothing writes once read: a turn kept holds its body, not+// a copy of it.+func prevInput(raw json.RawMessage) (all, fresh []json.RawMessage) {+ raw = bytes.TrimSpace(raw)+ if len(raw) == 0 {+ return nil, nil+ }+ if raw[0] == '"' {+ var s string+ if json.Unmarshal(raw, &s) != nil {+ return nil, nil+ }+ m, _ := json.Marshal(map[string]any{"type": "message", "role": "user", "content": s})+ return []json.RawMessage{m}, []json.RawMessage{m}+ }+ items, ok := inputList(raw)+ if !ok || raw[0] != '[' {+ return nil, nil+ }+ if items == nil {+ items = []json.RawMessage{}+ }+ return items, items+}++// prevReply is a Responses reply's id and output items, a stream's or a+// whole body's; ok false for one that didn't complete.+func prevReply(ctype string, reply []byte) (id string, output []json.RawMessage, ok bool) {+ var resp json.RawMessage+ if strings.Contains(ctype, "event-stream") || bytes.HasPrefix(reply, []byte("event:")) || bytes.HasPrefix(reply, []byte("data:")) {+ items := map[int]json.RawMessage{}+ readSSE(bytes.NewReader(reply), func(event, data string) error {+ var e struct {+ Type string `json:"type"`+ Response json.RawMessage `json:"response"`+ Item json.RawMessage `json:"item"`+ OutputIndex int `json:"output_index"`+ }+ if json.Unmarshal([]byte(data), &e) != nil {+ return nil+ }+ if e.Type == "" {+ e.Type = event+ }+ switch e.Type {+ case "response.output_item.done":+ if len(e.Item) > 0 {+ items[e.OutputIndex] = e.Item+ }+ case "response.completed", "response.incomplete":+ resp = e.Response+ }+ return nil+ })+ if len(resp) == 0 {+ return "", nil, false+ }+ resp = withStreamedOutput(resp, items)+ } else {+ resp = reply+ }+ var r struct {+ ID string `json:"id"`+ Object string `json:"object"`+ Status string `json:"status"`+ Output []json.RawMessage `json:"output"`+ }+ if json.Unmarshal(resp, &r) != nil || r.ID == "" || r.Status == "failed" || r.Status == "cancelled" {+ return "", nil, false+ }+ return r.ID, r.Output, true+}++// prevDelta is body, a Responses request magpie gave its conversation+// whole, with the new input alone after the id p keeps the previous+// response under, when p is the provider and key that answered it and+// keeps what it is sent. The tries' changes to the input are kept: the+// new items are the body's own last ones, and only when the body still+// has as many items as magpie gave it.+func prevDelta(r *http.Request, p provider.Provider, body []byte) ([]byte, bool) {+ a := prevAskOf(r.Context())+ if a == nil || !a.known || a.upstreamID == "" || a.route != prevRoute(p) || !prevKeptBy(p) {+ return body, false+ }+ var q map[string]json.RawMessage+ if decodeJSONView(body, &q) != nil {+ return body, false+ }+ var store bool+ if json.Unmarshal(q["store"], &store) == nil && !store {+ return body, false+ }+ items, ok := inputList(q["input"])+ if !ok || len(items) != a.total || a.fresh > len(items) {+ return body, false+ }+ in, err := json.Marshal(items[len(items)-a.fresh:])+ if err != nil {+ return body, false+ }+ id, _ := json.Marshal(a.upstreamID)+ out := make(map[string]json.RawMessage, len(q)+1)+ for k, v := range q {+ out[k] = v+ }+ out["input"], out["previous_response_id"] = in, id+ nb, err := json.Marshal(out)+ if err != nil {+ return body, false+ }+ return nb, true+}++// prevRefused is an upstream's answer that it doesn't hold the previous+// response, or takes no such field: the conversation is sent whole.+func prevRefused(status int, b []byte) bool {+ if !badRequest(status) && status != http.StatusNotFound {+ return false+ }+ s := strings.ToLower(string(b))+ return strings.Contains(s, "previous_response") || strings.Contains(s, "previousresponse")+}++// storeOff is whether a Responses body says store false.+func storeOff(body []byte) bool {+ v := gjson.GetBytes(body, "store")+ return v.Exists() && v.Type == gjson.False+}diff --git a/internal/gateway/prev_response_test.go b/internal/gateway/prev_response_test.gonew file mode 100644index 000000000..755ad0253--- /dev/null+++ b/internal/gateway/prev_response_test.go@@ -0,0 +1,263 @@+package gateway++import (+ "context"+ "encoding/json"+ "fmt"+ "io"+ "net/http"+ "net/http/httptest"+ "strings"+ "sync"+ "sync/atomic"+ "testing"+ "time"++ "github.com/yetone/magpie/internal/access"+ "github.com/yetone/magpie/internal/provider"+ "github.com/yetone/magpie/internal/settings"+)++// prevEnv is a sandbox with one provider, fresh responses kept, and an+// upstream that records every request and answers with answer.+type prevUp struct {+ mu sync.Mutex+ got [][]byte+ srv *httptest.Server+ next func(n int, body []byte) (int, string, string) // status, content type, body+}++func (u *prevUp) last() map[string]any {+ u.mu.Lock()+ defer u.mu.Unlock()+ var q map[string]any+ json.Unmarshal(u.got[len(u.got)-1], &q)+ return q+}++func (u *prevUp) calls() int {+ u.mu.Lock()+ defer u.mu.Unlock()+ return len(u.got)+}++func newPrevUp(t *testing.T, next func(n int, body []byte) (int, string, string)) *prevUp {+ u := &prevUp{next: next}+ u.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ b, _ := io.ReadAll(r.Body)+ u.mu.Lock()+ u.got = append(u.got, b)+ n := len(u.got)+ u.mu.Unlock()+ code, ct, reply := u.next(n, b)+ w.Header().Set("Content-Type", ct)+ w.WriteHeader(code)+ io.WriteString(w, reply)+ }))+ t.Cleanup(u.srv.Close)+ return u+}++func prevSandbox(t *testing.T, p provider.Provider) *Server {+ t.Helper()+ t.Setenv("XDG_CONFIG_HOME", t.TempDir())+ t.Setenv("XDG_CACHE_HOME", t.TempDir())+ old := prevResponses+ prevResponses = newPrevStore(prevCap, prevTTL)+ t.Cleanup(func() { prevResponses = old })+ if err := provider.Save(p); err != nil {+ t.Fatal(err)+ }+ return New()+}++// chatSSEReply is a Chat stream that says text.+func chatSSEReply(text string) string {+ c, _ := json.Marshal(text)+ return "data: {\"id\":\"c1\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{\"content\":" + string(c) + "},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":7,\"completion_tokens\":1}}\n\ndata: [DONE]\n\n"+}++func askResponses(t *testing.T, s *Server, keyID string, body string) *httptest.ResponseRecorder {+ t.Helper()+ ctx := context.Background()+ if keyID != "" {+ ctx = access.WithIdentity(ctx, access.Identity{KeyID: keyID, KeyName: keyID})+ }+ rec := httptest.NewRecorder()+ req := httptest.NewRequest("POST", "/v1/responses", strings.NewReader(body)).WithContext(ctx)+ req.RemoteAddr = "127.0.0.1:5555"+ s.Handler().ServeHTTP(rec, req)+ return rec+}++func responseID(t *testing.T, rec *httptest.ResponseRecorder) string {+ t.Helper()+ if rec.Code != 200 {+ t.Fatalf("%d %s", rec.Code, rec.Body.String())+ }+ var r struct {+ ID string `json:"id"`+ }+ if json.Unmarshal(rec.Body.Bytes(), &r) != nil || r.ID == "" {+ t.Fatalf("no id in %s", rec.Body.String())+ }+ return r.ID+}++// A client that sends only the new input after previous_response_id+// (kevinzhow on X) has the conversation given back whole, here to a+// provider on Chat: the model sees the earlier turn and its own answer.+func TestPreviousResponseGivenWholeToChat(t *testing.T) {+ up := newPrevUp(t, func(n int, _ []byte) (int, string, string) {+ return 200, "text/event-stream", chatSSEReply(fmt.Sprintf("answer %d", n))+ })+ s := prevSandbox(t, provider.Provider{ID: "fake", Name: "Fake", Key: "k", Models: []string{"m1"}, Chat: up.srv.URL + "/v1"})+ id := responseID(t, askResponses(t, s, "", `{"model":"fake/m1","instructions":"be brief","input":"my name is Ada"}`))+ id2 := responseID(t, askResponses(t, s, "", `{"model":"fake/m1","previous_response_id":"`+id+`","input":[{"role":"user","content":"what is my name?"}]}`))+ sent, _ := json.Marshal(up.last()["messages"])+ for _, want := range []string{"my name is Ada", "answer 1", "what is my name?"} {+ if !strings.Contains(string(sent), want) {+ t.Fatalf("turn 2 went without %q: %s", want, sent)+ }+ }+ // instructions go with the request that gives them, as OpenAI does+ if strings.Contains(string(sent), "be brief") {+ t.Fatalf("turn 1's instructions carried: %s", sent)+ }+ // and a third turn after the second has all of it+ responseID(t, askResponses(t, s, "", `{"model":"fake/m1","previous_response_id":"`+id2+`","input":"and again?"}`))+ sent, _ = json.Marshal(up.last()["messages"])+ for _, want := range []string{"my name is Ada", "answer 1", "what is my name?", "answer 2", "and again?"} {+ if !strings.Contains(string(sent), want) {+ t.Fatalf("turn 3 went without %q: %s", want, sent)+ }+ }+}++// An id magpie doesn't hold, to a provider it would be translated for,+// is OpenAI's own 400; so is one another gateway key's response, and one+// the client asked not be kept (store false).+func TestPreviousResponseNotFound(t *testing.T) {+ up := newPrevUp(t, func(int, []byte) (int, string, string) {+ return 200, "text/event-stream", chatSSEReply("ok")+ })+ s := prevSandbox(t, provider.Provider{ID: "fake", Name: "Fake", Key: "k", Models: []string{"m1"}, Chat: up.srv.URL + "/v1"})+ notFound := func(rec *httptest.ResponseRecorder, id string) {+ t.Helper()+ var e struct {+ Error struct{ Message, Type, Param, Code string }+ }+ json.Unmarshal(rec.Body.Bytes(), &e)+ if rec.Code != 400 || e.Error.Code != "previous_response_not_found" || e.Error.Param != "previous_response_id" || e.Error.Type != "invalid_request_error" || !strings.Contains(e.Error.Message, id) {+ t.Fatalf("%d %s", rec.Code, rec.Body.String())+ }+ }+ notFound(askResponses(t, s, "", `{"model":"fake/m1","previous_response_id":"resp_nowhere","input":"hi"}`), "resp_nowhere")+ if up.calls() != 0 {+ t.Fatalf("sent upstream without its history: %d", up.calls())+ }+ a := responseID(t, askResponses(t, s, "alice", `{"model":"fake/m1","input":"secret plan"}`))+ notFound(askResponses(t, s, "bob", `{"model":"fake/m1","previous_response_id":"`+a+`","input":"what was it?"}`), a)+ notFound(askResponses(t, s, "", `{"model":"fake/m1","previous_response_id":"`+a+`","input":"what was it?"}`), a)+ responseID(t, askResponses(t, s, "alice", `{"model":"fake/m1","previous_response_id":"`+a+`","input":"what was it?"}`))+ off := responseID(t, askResponses(t, s, "", `{"model":"fake/m1","store":false,"input":"forget me"}`))+ notFound(askResponses(t, s, "", `{"model":"fake/m1","previous_response_id":"`+off+`","input":"again"}`), off)+}++// An id magpie doesn't hold, to a provider on Responses the request is+// relayed to, goes as it came: a response the provider made before+// magpie restarted is still its own to find.+func TestUnknownPreviousResponseRelayedAsSent(t *testing.T) {+ up := newPrevUp(t, func(int, []byte) (int, string, string) {+ return 200, "application/json", `{"id":"resp_up","object":"response","status":"completed","output":[{"type":"message","role":"assistant","content":[{"type":"output_text","text":"ok"}]}],"usage":{"input_tokens":3,"output_tokens":1}}`+ })+ s := prevSandbox(t, provider.Provider{ID: "fake", Name: "Fake", Key: "k", Models: []string{"m1"}, Responses: up.srv.URL + "/v1"})+ responseID(t, askResponses(t, s, "", `{"model":"fake/m1","previous_response_id":"resp_before","input":"hi"}`))+ if got := up.last()["previous_response_id"]; got != "resp_before" {+ t.Fatalf("sent %v", got)+ }+}++// A provider that keeps responses (OpenAI's, Ark's) and answered the one+// named is sent the new input alone after its own id: the images it holds+// aren't sent again. One that no longer holds it is sent the whole.+func TestPreviousResponseKeptUpstreamSentNewInputOnly(t *testing.T) {+ defer func(old func(provider.Provider) bool) { prevKeptBy = old }(prevKeptBy)+ prevKeptBy = func(p provider.Provider) bool { return p.ID == "fake" }+ var forget atomic.Bool+ up := newPrevUp(t, func(n int, body []byte) (int, string, string) {+ if forget.Load() && strings.Contains(string(body), "previous_response_id") {+ return 400, "application/json", `{"error":{"message":"Previous response with id 'resp_up1' not found.","type":"invalid_request_error","param":"previous_response_id","code":"previous_response_not_found"}}`+ }+ return 200, "application/json", fmt.Sprintf(`{"id":"resp_up%d","object":"response","status":"completed","output":[{"type":"message","id":"msg_%d","role":"assistant","content":[{"type":"output_text","text":"answer %d"}]}],"usage":{"input_tokens":3,"output_tokens":1}}`, n, n, n)+ })+ s := prevSandbox(t, provider.Provider{ID: "fake", Name: "Fake", Key: "k", Models: []string{"m1"}, Responses: up.srv.URL + "/v1"})+ img := `{"type":"message","role":"user","content":[{"type":"input_image","image_url":"data:image/png;base64,` + strings.Repeat("QUFB", 50000) + `"},{"type":"input_text","text":"what is this?"}]}`+ id := responseID(t, askResponses(t, s, "", `{"model":"fake/m1","input":[`+img+`]}`))+ if id != "resp_up1" {+ t.Fatalf("client given %s", id)+ }+ rec := askResponses(t, s, "", `{"model":"fake/m1","previous_response_id":"`+id+`","input":"and now?"}`)+ if responseID(t, rec) != "resp_up2" {+ t.Fatal(rec.Body.String())+ }+ q := up.last()+ in, _ := json.Marshal(q["input"])+ up.mu.Lock()+ sent := len(up.got[1])+ up.mu.Unlock()+ if q["previous_response_id"] != "resp_up1" || strings.Contains(string(in), "QUFB") || !strings.Contains(string(in), "and now?") || sent > 2000 {+ t.Fatalf("sent %d bytes: prev %v input %s", sent, q["previous_response_id"], in)+ }+ // the upstream lost it: asked again with the conversation whole+ forget.Store(true)+ responseID(t, askResponses(t, s, "", `{"model":"fake/m1","previous_response_id":"resp_up2","input":"once more"}`))+ q = up.last()+ in, _ = json.Marshal(q["input"])+ if _, has := q["previous_response_id"]; has || !strings.Contains(string(in), "QUFB") || !strings.Contains(string(in), "answer 1") || !strings.Contains(string(in), "and now?") || !strings.Contains(string(in), "once more") {+ t.Fatalf("whole: prev %v input %.300s", q["previous_response_id"], in)+ }+}++// What a response named brings back is masked as the client's own words+// are: a key in an earlier turn never reaches the vendor.+func TestPreviousResponseMasked(t *testing.T) {+ const key = "sk-proj-abcdEFGH1234ijklMNOP5678qrst"+ up := newPrevUp(t, func(int, []byte) (int, string, string) {+ return 200, "text/event-stream", chatSSEReply("noted")+ })+ s := prevSandbox(t, provider.Provider{ID: "fake", Name: "Fake", Key: "k", Models: []string{"m1"}, Chat: up.srv.URL + "/v1"})+ if err := settings.Save(settings.Settings{Redact: true}); err != nil {+ t.Fatal(err)+ }+ id := responseID(t, askResponses(t, s, "", `{"model":"fake/m1","input":"my key is `+key+`"}`))+ responseID(t, askResponses(t, s, "", `{"model":"fake/m1","previous_response_id":"`+id+`","input":"what was it?"}`))+ sent, _ := json.Marshal(up.last())+ if strings.Contains(string(sent), key) || !strings.Contains(string(sent), "my key is {{") {+ t.Fatalf("sent %s", sent)+ }+}++// The kept responses are bounded by bytes and time, the oldest out first.+func TestPrevStoreBounded(t *testing.T) {+ st := newPrevStore(100, prevTTL)+ item := json.RawMessage(strings.Repeat("x", 40))+ for i := range 4 {+ st.put(&prevTurn{key: prevKey{"s", fmt.Sprint(i)}, items: []json.RawMessage{item}, own: 40})+ }+ if _, ok := st.get(prevKey{"s", "0"}); ok {+ t.Fatal("oldest kept past the cap")+ }+ if _, ok := st.get(prevKey{"s", "3"}); !ok || st.size > 100 {+ t.Fatalf("newest gone, size %d", st.size)+ }+ st.put(&prevTurn{key: prevKey{"s", "big"}, own: 60})+ if _, ok := st.get(prevKey{"s", "big"}); ok {+ t.Fatal("a turn past half the cap kept")+ }+ now := st.now()+ st.now = func() time.Time { return now.Add(prevTTL + 1) }+ if _, ok := st.get(prevKey{"s", "3"}); ok {+ t.Fatal("kept past its time")+ }+}