Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
provider: every request to a Remote magpie carries the provider's Headers; its key always goes as x-api-key (Yirasumi)
yetone66b5e4a844cd
diff --git a/docs/subsystems/providers-accounts.md b/docs/subsystems/providers-accounts.mdindex 8903211bc..abf2172b0 100644--- a/docs/subsystems/providers-accounts.md+++ b/docs/subsystems/providers-accounts.md@@ -23,6 +23,7 @@ keys and accounts per request is in [Gateway routing and fallback](gateway-routi| Zhipu's 200 refusals | Zhipu's and Z.ai's `/api/v1` (the GLM Coding Plan's Responses API, from their Codex pages) and `/api/monitor` answer a refusal 200 in the platform's envelope, `{"code":1000,"msg":"身份验证失败。","success":false}`, where their OpenAI and Anthropic endpoints answer the same key 401 (@xanahopper on X: a refused key tested green on the Responses row only). `ZhipuEnveloped` turns a 2xx JSON body from `open.bigmodel.cn` or `api.z.ai` with `success: false` into `{"error":{"code","message"}}` at the status Zhipu's error-code table gives (1000–1005 → 401, 1113 and 1302–1399 → 429, 1220 → 403, 1210–1229, 1261 and 1301 → 400, anything else → 502), in the provider test's `probe` (endpoint tests, model tests, Detect APIs) and the gateway's `forwardOnce`. A stream, a compressed body, a body over 64 KB and every other host pass as they came. The plan's quota card for a key refused there (1000–1005, or a 401) says `<Zhipu or Z.ai> didn't take this key (<their words>): …` with the vendor's API-keys page (`zhipuRefused`), not asked again as a team's key, and the GUI shows it as that (`quotaError`) rather than "Allowance unavailable". | [`zhipu_envelope.go`](../../internal/provider/zhipu_envelope.go), [`planquota.go`](../../internal/provider/planquota.go), [`test.go`](../../internal/provider/test.go) || Extra body | `ExtraBody` (`extraBody` in providers.json, the editor's *Extra body*, #1604) is a JSON object whose fields `WithExtraBody` adds at the top level of every JSON body sent to the provider: Messages, Chat, Responses and Gemini, relayed as they came or translated, in the gateway's `forwardOnce` and in Test and Detect (`tested`). A field the body has already is kept; a name written `"!name"` replaces it. Values are put in as written, objects whole, not merged. `${sessionId}` in any string, at any depth, becomes the request's session: the agent's session header (`sessionOf`), else the id `conversationID` derives from the first user message (`magpie-` and its hash), so it stays the same through one conversation; Test and Detect send `magpie-test-` and 12 random letters, a new one each time. `${threadId}` is the thread the agent names within that session: Codex's `thread_id` (its turn metadata or `thread-id` header, `threadOf`), which a subagent has of its own while it shares its lead's session; read only when the provider's headers or extraBody name it (`AsksThread`). No other agent names a thread, and there `${threadId}` is the session's id. `model`, `stream` and the conversation (`messages`, `input`, `contents`, `system`, `systemInstruction`, `system_instruction`, `instructions`, `tools`, `ExtraBodyReserved`) are magpie's: Save refuses them with or without `!` (`ExtraBodyOK`), the editor says so as they are typed, and one written into providers.json by hand is dropped on read (`cleanExtraBody`). Not sent with it: `count_tokens`, embeddings, images, video and System One's decision model (other APIs), and a signed-in account's requests, except a ChatGPT (Codex) account's (magpie-releases#18): its requests are built as Codex CLI's (`codexBody`) and take the fields Codex CLI adds by choice, such as `access_programs` (OpenAI's Daybreak), while `store`, `tool_choice` and `include`, which the backend needs as `codexBody` sets them, can't be set (`codexOwned`). Its extraBody is written into providers.json by hand under the account's entry and kept onto the account by `withPicks`; the editor has no box for it. A client's own top-level `access_programs` on Responses or Chat is carried to a Responses build as it came (`Request.AccessPrograms`), so a request translated (a Codex account is streamed, so a whole Responses request is made again) keeps it as a relayed one does. It reaches only the upstreams Codex itself sends it to (`keepsAccessPrograms`, #3172, see [gateway routing](gateway-routing.md)); it is taken out before the extraBody merge, so an `access_programs` the user puts in a provider's extraBody still goes. One object goes to every API; fields for one API alone are not a thing yet. A `thinking` budget it puts in on an Anthropic endpoint is fitted to a Claude that thinks only adaptively after the merge, as an agent's is (`adaptiveThinking`, `TestExtraBodyBudgetFittedToAdaptiveModel`). | [`extrabody.go`](../../internal/provider/extrabody.go), [`gateway.go`](../../internal/gateway/gateway.go) || Headers' conversation ids | `${sessionId}` and `${threadId}` in a value of the provider's `Headers` (Axel Burks on Discord: a gateway in front of several Azure accounts routes one conversation to one account by a session header) are the ids extraBody gets, on every request `Sign` signs: the gateway carries the conversation to it in the request's context (`WithConversation`), on every API and `count_tokens` too. A request that is no turn of a conversation (a model list, a balance, an embedding, image or video, System One's decision model, a search) gets an id of its own, `magpie-` and 12 random letters, new each time (`OwnConversation`); Test and Detect's is their `magpie-test-` id, the same in headers and body. A literal placeholder is never sent. In a header the ids go without control characters (`headerSafe`): an agent's session id with CR or LF in it would split the header, or Go would fail the request. A value with neither placeholder goes as written (`SentHeaders`). | [`conversation.go`](../../internal/provider/conversation.go), [`account.go`](../../internal/provider/account.go), [`gateway.go`](../../internal/gateway/gateway.go), [`codex_resume.go`](../../internal/gateway/codex_resume.go) |+| A remote magpie's headers | Every request magpie makes to a Remote magpie carries the provider's `Headers` (Yirasumi on Discord: the Usage page's read went without the header a proxy in front of the remote asks for): its API requests and model list as any provider's (`Sign`, `listHeaders`), and its cards, a card's refresh, its quota history and each gateway key's limit (`signRemote`), the last four as no turn of a conversation (`${sessionId}` an id of its own). The key always reaches the remote as `x-api-key` (`remoteKey`), which a gateway reads beside `Authorization`: a user's `Authorization` (a proxy's own scheme) goes in place of magpie's `Bearer` copy without taking the key away, and a user's `x-api-key` is never sent to a remote magpie, so a key's limit is read with that key. A user's `Authorization` that is itself another gateway key of that magpie is read first there, as on any request. | [`remote_quotas.go`](../../internal/provider/remote_quotas.go), [`remote_key_limits.go`](../../internal/provider/remote_key_limits.go), [`remote_magpie.go`](../../internal/provider/remote_magpie.go), [`account.go`](../../internal/provider/account.go) || Reply headers | `ReplyHeaders` (`replyHeaders` in providers.json, the editor's *Reply headers* under Advanced, a comma-separated list, #4145) names headers of the vendor's reply the gateway passes on to the agent, matched whatever their case and saved canonical (`cleanReplyHeaders`). `DefaultReplyHeaders` (`Request-Id`, `X-Request-Id`, `X-Oneapi-Request-Id`) go from every provider without being listed: opaque request ids, under names magpie sets nothing of its own. Never passed on, listed or not (`ReplyHeaderPassable`): the reply's framing (`Content-*`, `Transfer-Encoding`, hop-by-hop), cookies and sign-ins (`Set-Cookie`, `Authorization`, `WWW-Authenticate`, `Proxy-*`), CORS (`Access-Control-*`), `Sec-*`, redirects and browser policy (`Location`, `Strict-Transport-Security`, `Content-Security-Policy`…), `Retry-After`/`Cache-Control`/`Date`/`Server`/`Vary` (magpie writes those) and `X-Magpie-*` (the gateway reads them as notes of the try). Save refuses such a name (`ReplyHeadersOK`), the editor says so before the save (`replyHeadersOf`, its list held to Go's by `TestReplyHeadersNeverAsTheProvidersAre`), and one written into providers.json by hand is dropped on read. A preset's provider and a signed-in account's requests take it too (`withPicks`). How the gateway carries them is in [gateway routing](gateway-routing.md). | [`replyheaders.go`](../../internal/provider/replyheaders.go), [`reply_headers.go`](../../internal/gateway/reply_headers.go) || Native compaction | `NativeCompaction` (`nativeCompaction` in providers.json, the editor's *Compaction* row, `magpie provider set <id> compaction=native\|magpie`; #1099) has Codex's compaction asked of the provider as Codex sent it and its sealed item returned unchanged. Off by default. Only a provider whose wire carries it can be set so (`CarriesCompaction`: a Responses API, a ChatGPT account, GitHub Copilot; not another agent's sign-in or a plugin's), and `CompactsOn(model)` also needs the model asked on Responses. `SetNativeCompaction`, the CLI and the GUI's save refuse it elsewhere. An account keeps it through `Save`'s picks (`withPicks`), and `AddCopy` carries it. The gateway's half is in [gateway-routing](gateway-routing.md). | [`native_compaction.go`](../../internal/provider/native_compaction.go) || providers.json | `Path` is `providers.json` in magpie's config folder. A read that fails returns `ErrUnreadable` and is never treated as empty. `store` writes atomically through a rename and drops the cached catalog (`catalog.Touched`). | [`provider.go`](../../internal/provider/provider.go) |diff --git a/internal/provider/account.go b/internal/provider/account.goindex f228c8ef2..7d2e320e3 100644--- a/internal/provider/account.go+++ b/internal/provider/account.go@@ -241,6 +241,12 @@ func (p Provider) Sign(ctx context.Context, req *http.Request, proto Protocol, b}catalog.PutUserHeader(req.Header, k, v)}+ // a remote magpie is always sent its key as x-api-key, which no header+ // of the user's replaces: their Authorization (a proxy's) goes on+ // without taking the key away (remoteKey)+ if p.IsRemoteMagpie() && p.Key != "" {+ remoteKey(req.Header, p.Key)+ }return nil}diff --git a/internal/provider/remote_headers_test.go b/internal/provider/remote_headers_test.gonew file mode 100644index 000000000..0dfdf5b44--- /dev/null+++ b/internal/provider/remote_headers_test.go@@ -0,0 +1,124 @@+package provider++import (+ "bytes"+ "context"+ "encoding/json"+ "net/http"+ "regexp"+ "strings"+ "sync"+ "testing"+)++// headerPeer is a remote magpie behind a proxy that wants its own+// headers: it answers every request of magpie's own (cards, a refresh,+// the quota history, a key's limit, the model list) and keeps each one's+// headers by its path.+type headerPeer struct {+ sync.Mutex+ seen map[string][]http.Header+}++func (f *headerPeer) ServeHTTP(w http.ResponseWriter, r *http.Request) {+ f.Lock()+ f.seen[r.Method+" "+r.URL.Path] = append(f.seen[r.Method+" "+r.URL.Path], r.Header.Clone())+ f.Unlock()+ switch r.URL.Path {+ case RemoteKeyLimitPath:+ w.Write([]byte(`{"limited":false,"key":"k"}`))+ case "/v1/magpie/quotas/history":+ json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": []QuotaHistory{{Provider: "codex"}}})+ case "/v1/models":+ w.Write([]byte(`{"object":"list","data":[{"id":"gpt-6","object":"model"}]}`))+ default:+ json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": []SubscriptionQuota{{Provider: "codex", Name: "Codex", Kind: "subscription"}}})+ }+}++// Yirasumi on Discord: a Remote magpie behind a proxy that asks for a+// header of its own answered the chat, but the Usage page's read of its+// cards went without the provider's Headers. Every request magpie makes+// to a remote magpie now carries them, ${sessionId} as an id of its own,+// and the key still reaches the remote as x-api-key: a user's+// Authorization (the proxy's) goes on in place of magpie's Bearer, and a+// user's x-api-key never replaces the key.+func TestRemoteMagpieRequestsCarryHeaders(t *testing.T) {+ for _, userAuth := range []bool{false, true} {+ t.Run(map[bool]string{false: "own auth", true: "user's Authorization"}[userAuth], func(t *testing.T) {+ peer := &headerPeer{seen: map[string][]http.Header{}}+ remoteQuotaHome(t, peer)+ p, err := Find("office")+ if err != nil {+ t.Fatal(err)+ }+ p.Headers = map[string]string{"X-Proxy-Token": "tok", "X-Session": "${sessionId}", "X-Api-Key": "not-the-key"}+ if userAuth {+ p.Headers["Authorization"] = "Basic cHJveHk6cHc="+ }+ p.Keys = []KeyAccount{{Name: "spare", Key: "sk-magpie-spare"}}+ if err := Save(*p); err != nil {+ t.Fatal(err)+ }+ ForgetRemoteCardsForTest()+ if p, err = Find("office"); err != nil {+ t.Fatal(err)+ }+ ctx := context.Background()+ RemoteCards(ctx)+ RefreshUsage(ctx, "office/codex", "")+ RemoteQuotaHistories(ctx, "7")+ if _, err := p.Fetch(ctx); err != nil {+ t.Fatalf("model list: %v", err)+ }+ req, _ := http.NewRequest(http.MethodPost, p.Chat+"/chat/completions", bytes.NewReader([]byte(`{}`)))+ if err := p.Sign(ctx, req, Chat, []byte(`{}`)); err != nil {+ t.Fatal(err)+ }+ res, err := p.Do(http.DefaultClient, req)+ if err != nil {+ t.Fatal(err)+ }+ res.Body.Close()++ own := regexp.MustCompile(`^magpie-[a-z0-9]{12}$`)+ peer.Lock()+ defer peer.Unlock()+ for _, path := range []string{+ "GET " + RemoteCardsPath, "POST " + RemoteRefreshPath, "GET /v1/magpie/quotas/history",+ "GET " + RemoteKeyLimitPath, "GET /v1/models", "POST /v1/chat/completions",+ } {+ hs := peer.seen[path]+ if len(hs) == 0 {+ t.Errorf("%s: never asked", path)+ continue+ }+ keys := map[string]bool{}+ for _, h := range hs {+ if h.Get("X-Proxy-Token") != "tok" {+ t.Errorf("%s: X-Proxy-Token %q", path, h.Get("X-Proxy-Token"))+ }+ if s := h.Get("X-Session"); !own.MatchString(s) {+ t.Errorf("%s: X-Session %q, want an id of its own", path, s)+ }+ if v := h.Values("X-Api-Key"); len(v) != 1 || !strings.HasPrefix(v[0], "sk-magpie-") {+ t.Errorf("%s: x-api-key %q, want magpie's key", path, v)+ } else {+ keys[v[0]] = true+ }+ want := "Bearer " + h.Get("X-Api-Key")+ if userAuth {+ want = "Basic cHJveHk6cHc="+ }+ if a := h.Values("Authorization"); len(a) != 1 || a[0] != want {+ t.Errorf("%s: Authorization %q, want %q", path, a, want)+ }+ }+ // each key's limit is read with that key+ if path == "GET "+RemoteKeyLimitPath && (!keys["sk-magpie-office"] || !keys["sk-magpie-spare"]) {+ t.Errorf("key limits read with %v", keys)+ }+ }+ })+ }+}diff --git a/internal/provider/remote_key_limits.go b/internal/provider/remote_key_limits.goindex 9f2d8efb6..5770a8b53 100644--- a/internal/provider/remote_key_limits.go+++ b/internal/provider/remote_key_limits.go@@ -238,8 +238,8 @@ func askRemoteKeyLimit(ctx context.Context, p Provider, key string) (*remoteKeyAif err != nil {return nil, err}- req.Header.Set("Authorization", "Bearer "+key)req.Header.Set("Accept", "application/json")+ p.signRemote(req, key)res, err := p.Do(http.DefaultClient, req)if err != nil {return nil, fmt.Errorf("%s %s: %w", errRemoteUnreachable, remoteName(p), err)diff --git a/internal/provider/remote_magpie.go b/internal/provider/remote_magpie.goindex 6f08794cf..9ff7f03cc 100644--- a/internal/provider/remote_magpie.go+++ b/internal/provider/remote_magpie.go@@ -109,6 +109,7 @@ func (p *Provider) remoteMagpieEndpoints() {// listHeaders are the headers p's model list is asked with: a remote// magpie is asked for its image, video and decision models too, and its prices.+// Its key goes as x-api-key whatever the user's headers say (remoteKey).// A list is no turn of a conversation: ${sessionId} and ${threadId} in a// value are an id of the list's own (conversation.go).func (p Provider) listHeaders() map[string]string {@@ -120,6 +121,11 @@ func (p Provider) listHeaders() map[string]string {if h == nil {h = map[string]string{}}+ for k := range h {+ if strings.EqualFold(k, "x-api-key") {+ delete(h, k)+ }+ }h[DrawersHeader] = "1"h[VideomakersHeader] = "1"h[PricesHeader] = "1"diff --git a/internal/provider/remote_quotas.go b/internal/provider/remote_quotas.goindex e83ac1ad9..37de272f5 100644--- a/internal/provider/remote_quotas.go+++ b/internal/provider/remote_quotas.go@@ -12,6 +12,8 @@ import ("strings""sync""time"++ "github.com/yetone/magpie/internal/catalog")// A remote magpie's quotas (莫 on Discord): a computer whose providers are@@ -360,6 +362,35 @@ func askRemote(ctx context.Context, p Provider, method, path string) ([]Subscripreturn cards, nil}+// signRemote puts on req, a request of magpie's own to the remote magpie+// p (its cards, a refresh, its quota history, a key's limit), key and p's+// Headers, as its model list and its API requests carry them (Yirasumi on+// Discord: the Usage page's read of the remote went without the headers a+// proxy in front of it asks for). It is no turn of a conversation:+// ${sessionId} and ${threadId} are an id of its own (conversation.go).+// The key goes as Authorization: Bearer, which a user's Authorization+// replaces (a proxy's own scheme), and always as x-api-key (remoteKey).+func (p Provider) signRemote(req *http.Request, key string) {+ req.Header.Set("Authorization", "Bearer "+key)+ for k, v := range p.SentHeaders(context.Background()) {+ catalog.PutUserHeader(req.Header, k, v)+ }+ remoteKey(req.Header, key)+}++// remoteKey writes key as x-api-key, in place of the header in any case:+// to a remote magpie that header is magpie's key, which a gateway reads+// beside Authorization, so a user's header (a proxy's Authorization) never+// leaves the request without it, and a key's limit is read with that key.+func remoteKey(h http.Header, key string) {+ for k := range h {+ if strings.EqualFold(k, "x-api-key") {+ delete(h, k)+ }+ }+ h.Set("x-api-key", key)+}+// remoteJSON asks p's gateway, with its key, for a {"data": …} list.func remoteJSON(ctx context.Context, p Provider, method, path string, dst any) error {ctx, cancel := context.WithTimeout(ctx, 8*time.Second)@@ -368,8 +399,8 @@ func remoteJSON(ctx context.Context, p Provider, method, path string, dst any) eif err != nil {return err}- req.Header.Set("Authorization", "Bearer "+p.Key)req.Header.Set("Accept", "application/json")+ p.signRemote(req, p.Key)res, err := p.Do(http.DefaultClient, req)if err != nil {return fmt.Errorf("%s %s: %w", errRemoteUnreachable, remoteName(p), err)