Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

gateway: magpie's own searches and image descriptions go without the agent's middleware (#3695)

JoeyWu-tech430e3050d86a
diff --git a/docs/subsystems/gateway-middleware.md b/docs/subsystems/gateway-middleware.md
index fbd09415f..713870ab0 100644
--- a/docs/subsystems/gateway-middleware.md
+++ b/docs/subsystems/gateway-middleware.md
@@ -29,7 +29,7 @@ plugins.json is the source of truth for which middleware is installed and on, in
Each middleware gets one runtime for the whole request, so its hooks share `ctx.state`. Runtimes come from a per-middleware `sync.Pool` and modules are compiled once per load. Compiling a plugin's module (imports kept inside the plugin) is [`internal/jsmod`](../../internal/jsmod/jsmod.go), which agent plugins ([Agent wiring](agent-wiring.md)) share.
-Only requests agents send go through `serveAgent`. magpie's own requests (thread titles, the router's classifier, the search and vision stand-ins) call `serve` directly. Middleware sees bodies before redaction, which still applies upstream, through the `redact` plugin's masker; masking asked for with no masker serving sends nothing ([Plugin runtime](plugin-runtime.md)).
+Only requests agents send go through `serveAgent`. magpie's own requests (thread titles, the router's classifier, the search and vision stand-ins) call `serve` directly. A search or an image's description asked while an agent's request is served is made on that request's context, and takes the request's middleware off it (`ownAsk`): the runtimes are the request's, which one goroutine may use at a time (moejs), and `Run` has no lock to hold callers to it, and these asks go beside it and each other, so images described at once ran `onProvider` on one runtime and the gateway died of `concurrent map read and map write`. Nor does the hooks' `ctx` describe them: a description's body is Chat and a search's Anthropic, whatever `ctx.protocol` names, and `ctx.model` and `ctx.stream` are the agent's (#3695, `TestImagesDescribedWithoutTheAgentsMiddleware`, `TestSearchesWithoutTheAgentsMiddleware`). Middleware sees bodies before redaction, which still applies upstream, through the `redact` plugin's masker; masking asked for with no masker serving sends nothing ([Plugin runtime](plugin-runtime.md)).
## Routings and usage rows
@@ -59,7 +59,7 @@ go test -tags nogui -run 'TestParseRouting|TestPluginListSaysMiddleware' ./inter
go test -tags nogui -run TestPluginRoutingKept ./internal/provider
go test -tags nogui -run TestOnServesStrategies ./plugins/middleware
node --test internal/gui/tests/routing-js.test.cjs
-go test -tags nogui -run 'TestMiddlewareThroughGateway|TestMiddlewareOnProviderPerTry' ./internal/gateway
+go test -tags nogui -run 'TestMiddlewareThroughGateway|TestMiddlewareOnProviderPerTry|WithoutTheAgentsMiddleware' ./internal/gateway
go test -tags nogui -run TestPluginListSaysMiddleware .
go test -tags nogui -run "TestSetOptionsShortName|TestBuiltinMarket" ./internal/plugin
go test -tags nogui -bench Event ./internal/middleware
diff --git a/internal/gateway/middleware.go b/internal/gateway/middleware.go
index 2f12640cb..c7978d87d 100644
--- a/internal/gateway/middleware.go
+++ b/internal/gateway/middleware.go
@@ -85,6 +85,20 @@ func triedOn(ctx context.Context, p provider.Provider, model string, body []byte
return body
}
+// ownAsk is ctx for a request magpie asks of its own gateway while it
+// serves an agent's (a web search, an image's description): the agent's
+// middleware isn't run on it, as on magpie's other own requests. Its
+// runtimes are the agent request's, which one goroutine may use at a time
+// (moejs), and Run has no lock to hold callers to it, while these asks go
+// beside the request and each other: images described at once ran
+// onProvider on one runtime, and the gateway died of "concurrent map read
+// and map write" (#3695). Nor does the hooks' ctx describe them: a
+// description's body is Chat and a search's Anthropic, whatever
+// ctx.protocol names, and ctx.model and ctx.stream are the agent's.
+func ownAsk(ctx context.Context) context.Context {
+ return context.WithValue(ctx, triedKey{}, []gwmw.Tried(nil))
+}
+
// middlewareWanted: plugins.json has a JS middleware on, so plugins/middleware
// is in use.
func middlewareWanted() bool { return middleware.On() > 0 }
diff --git a/internal/gateway/middleware_test.go b/internal/gateway/middleware_test.go
index 41c1431b7..cd52e2faf 100644
--- a/internal/gateway/middleware_test.go
+++ b/internal/gateway/middleware_test.go
@@ -1,7 +1,10 @@
package gateway
import (
+ "context"
+ "encoding/base64"
"encoding/json"
+ "fmt"
"io"
"net/http"
"net/http/httptest"
@@ -161,3 +164,161 @@ export function onProvider(body, ctx) {
t.Fatalf("state %+v", st)
}
}
+
+// onMiddleware puts a middleware of src on for t, alone in plugins.json.
+func onMiddleware(t *testing.T, src string) string {
+ t.Helper()
+ // the plugins' list written below reads as changed, and what that
+ // tells the plugins' hooks ends with the test
+ t.Cleanup(plugin.Settle)
+ mw := filepath.Join(t.TempDir(), "mark.middleware.js")
+ if err := os.WriteFile(mw, []byte(src), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ b, _ := json.Marshal(plugin.List{Plugins: []plugin.Entry{{Spec: mw}}})
+ os.MkdirAll(settings.Dir(), 0o755)
+ if err := os.WriteFile(filepath.Join(settings.Dir(), "plugins.json"), b, 0o644); err != nil {
+ t.Fatal(err)
+ }
+ middleware.Reload()
+ t.Cleanup(middleware.Reload)
+ return mw
+}
+
+// A request's images, each described by the model that sees, are magpie's
+// own asks and go without the agent's middleware. Up to four are asked at
+// once, and ran the request's onProvider on its one runtime from as many
+// goroutines: the gateway died of "concurrent map read and map write" in
+// moejs (#3695, aiirobyte). The agent's own try still goes through it.
+func TestImagesDescribedWithoutTheAgentsMiddleware(t *testing.T) {
+ s, u := eyed(t)
+ mw := onMiddleware(t, `
+export function onProvider(body, ctx) {
+ body.messages.push({role: "user", content: "onProvider for " + ctx.provider.model});
+ return body;
+}`)
+ var parts []string
+ for i := range visionParallel {
+ img := base64.StdEncoding.EncodeToString(fmt.Appendf(nil, "screenshot %d", i))
+ parts = append(parts, `{"type":"image_url","image_url":{"url":"data:image/png;base64,`+img+`"}}`)
+ }
+ rec := httptest.NewRecorder()
+ s.Handler().ServeHTTP(rec, httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(
+ `{"model":"probe/text","messages":[{"role":"user","content":[{"type":"text","text":"read"},`+strings.Join(parts, ",")+`]}]}`)))
+ if rec.Code != 200 {
+ t.Fatalf("%d %s", rec.Code, rec.Body.String())
+ }
+ eye := u.sent("eye")
+ if len(eye) != visionParallel {
+ t.Fatalf("%d images described, want %d", len(eye), visionParallel)
+ }
+ for _, b := range eye {
+ if strings.Contains(b, "onProvider for") {
+ t.Errorf("a description was asked through the agent's middleware: %s", b)
+ }
+ }
+ if text := u.sent("text"); len(text) != 1 || !strings.Contains(text[0], "onProvider for text") {
+ t.Fatalf("the agent's model was sent %q", text)
+ }
+ if st := middleware.States()[mw]; st.Calls != 1 || st.Failures != 0 {
+ t.Fatalf("state %+v, want onProvider called once, on the agent's try", st)
+ }
+}
+
+// The same for a web search magpie makes for a model that can't search:
+// it is asked beside the reply the agent is streamed through the
+// middleware's onEvent, so one search is enough to share the runtime, and
+// a round's searches beside each other (#3695). A hook that throws lets a
+// search through unmarked, so none may.
+func TestSearchesWithoutTheAgentsMiddleware(t *testing.T) {
+ fresh(t)
+ var mu sync.Mutex
+ var searched, asked []string
+ search := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ b, _ := io.ReadAll(r.Body)
+ switch r.URL.Path {
+ case "/v1/models":
+ io.WriteString(w, `{"data":[{"id":"claude-opus-5-5"},{"id":"claude-haiku-4-5"}]}`)
+ case "/v1/messages":
+ mu.Lock()
+ searched = append(searched, string(b))
+ mu.Unlock()
+ w.Header().Set("Content-Type", "application/json")
+ io.WriteString(w, `{"id":"s","type":"message","role":"assistant","model":"claude-haiku-4-5","content":[
+ {"type":"server_tool_use","id":"srv","name":"web_search","input":{"query":"go"}},
+ {"type":"web_search_tool_result","tool_use_id":"srv","content":[{"type":"web_search_result","title":"Go downloads","url":"https://go.dev/dl/"}]},
+ {"type":"text","text":"Found it."}],"stop_reason":"end_turn","usage":{"input_tokens":5,"output_tokens":5}}`)
+ default:
+ http.NotFound(w, r)
+ }
+ }))
+ defer search.Close()
+ model := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ b, _ := io.ReadAll(r.Body)
+ mu.Lock()
+ asked = append(asked, string(b))
+ n := len(asked)
+ mu.Unlock()
+ w.Header().Set("Content-Type", "text/event-stream")
+ if n == 1 {
+ // two searches in one round
+ io.WriteString(w, sse(`data: {"id":"c","choices":[{"index":0,"delta":{"role":"assistant","content":"Let me check."}}]}`,
+ `data: {"id":"c","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_1","type":"function","function":{"name":"web_search","arguments":"{\"query\":\"latest go\"}"}}]}}]}`,
+ `data: {"id":"c","choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":"call_2","type":"function","function":{"name":"web_search","arguments":"{\"query\":\"latest rust\"}"}}]}}]}`,
+ `data: {"id":"c","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}],"usage":{"prompt_tokens":30,"completion_tokens":7}}`,
+ `data: [DONE]`))
+ return
+ }
+ io.WriteString(w, sse(`data: {"id":"d","choices":[{"index":0,"delta":{"role":"assistant","content":"Both are out."}}]}`,
+ `data: {"id":"d","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":5}}`,
+ `data: [DONE]`))
+ }))
+ defer model.Close()
+ hosts := searchHosts[provider.Anthropic]
+ searchHosts[provider.Anthropic] = append(hosts, provider.HostOf(search.URL))
+ defer func() { searchHosts[provider.Anthropic] = hosts }()
+ for _, p := range []provider.Provider{
+ {ID: "srch", Name: "Search", Key: "k", Anthropic: search.URL},
+ {ID: "deep", Name: "Deep", Key: "k", Chat: model.URL, Models: []string{"deep-chat"}},
+ } {
+ if err := provider.Save(p); err != nil {
+ t.Fatal(err)
+ }
+ }
+ p, _ := provider.Find("srch")
+ if _, err := p.Fetch(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if sp, m, ok := searcher(); !ok || sp.ID != "srch" || m != "claude-haiku-4-5" {
+ t.Fatalf("searcher = %s %s %v", sp.ID, m, ok)
+ }
+ mw := onMiddleware(t, `
+export function onProvider(body, ctx) {
+ body.system = "onProvider for " + ctx.provider.model;
+ return body;
+}
+export const events = ["content_block_delta"];
+export function onEvent(ev) { return ev; }`)
+
+ code, out := post(t, "/v1/messages", `{"model":"deep/deep-chat","max_tokens":1000,"stream":true,"messages":[{"role":"user","content":"What are the latest Go and Rust?"}],
+ "tools":[{"type":"web_search_20250305","name":"web_search","max_uses":8}]}`)
+ if code != 200 || !strings.Contains(out, "Both are out.") {
+ t.Fatalf("%d %s", code, out)
+ }
+ mu.Lock()
+ defer mu.Unlock()
+ if len(searched) != 2 {
+ t.Fatalf("searched %d times, want 2", len(searched))
+ }
+ for _, b := range searched {
+ if strings.Contains(b, "onProvider for") {
+ t.Errorf("a search was asked through the agent's middleware: %s", b)
+ }
+ }
+ if len(asked) != 2 || !strings.Contains(asked[0], "onProvider for deep-chat") {
+ t.Fatalf("the agent's model was asked %d times, first with %s", len(asked), asked[0])
+ }
+ if st := middleware.States()[mw]; st.Failures != 0 {
+ t.Fatalf("state %+v, want no hook failed", st)
+ }
+}
diff --git a/internal/gateway/search.go b/internal/gateway/search.go
index 2203e6771..7215d01cd 100644
--- a/internal/gateway/search.go
+++ b/internal/gateway/search.go
@@ -916,7 +916,7 @@ var errNotSearched = fmt.Errorf("answered without searching the web: %w", errFou
// searching for its own models is sent none.
func (s *Server) searchWith(ctx context.Context, p provider.Provider, model, query, effort string) (string, []Hit, error) {
body, _ := json.Marshal(searchBody(p, model, query, effort))
- r, err := http.NewRequestWithContext(magpieChose(ctx), http.MethodPost, "http://magpie/v1/messages", nil)
+ r, err := http.NewRequestWithContext(ownAsk(magpieChose(ctx)), http.MethodPost, "http://magpie/v1/messages", nil)
if err != nil {
return "", nil, err
}
diff --git a/internal/gateway/vision.go b/internal/gateway/vision.go
index 6790cfc07..f44917618 100644
--- a/internal/gateway/vision.go
+++ b/internal/gateway/vision.go
@@ -381,7 +381,7 @@ func (s *Server) askVisionWith(ctx context.Context, model, src, system, text str
}
}
body, _ := json.Marshal(req)
- r, err := http.NewRequestWithContext(magpieChose(ctx), http.MethodPost, "http://magpie/v1/chat/completions", nil)
+ r, err := http.NewRequestWithContext(ownAsk(magpieChose(ctx)), http.MethodPost, "http://magpie/v1/chat/completions", nil)
if err != nil {
return "", err
}
diff --git a/site/public/docs/ja/plugins.html b/site/public/docs/ja/plugins.html
index 189051c40..af8d72e64 100644
--- a/site/public/docs/ja/plugins.html
+++ b/site/public/docs/ja/plugins.html
@@ -710,7 +710,7 @@ magpie plugin options model-map '{"mapping": {"fast": "deepseek/deepseek-chat"}}
<p><code>export const events = ["content_block_delta"]</code> と書くと、<code>onEvent</code> はその名前のイベントだけで呼ばれます(Anthropic Messages と Responses のイベントには名前があります)。Chat Completions と Gemini のイベントには名前がないので、それらでは <code>events</code> を書かないでください。1 つのイベントは応答の一部なので、1 つの単語が 2 つのイベントに分かれることがあります。<code>data: [DONE]</code> の行とコメントはフックを通らずにそのまま流れます。圧縮された応答もそのまま流れます。</p>
<p><code>ctx</code> には次のものがあります:<code>protocol</code>(<code>anthropic</code>・<code>chat</code>・<code>responses</code>・<code>gemini</code>)、エージェントが指定した <code>model</code> と <code>stream</code>、<code>path</code>、magpie が把握している場合の <code>agent</code>、<code>options</code>(そのプラグインの<a href="#middleware-options">オプション</a>)、そしてリクエストごとのオブジェクト <code>state</code>。</p>
<h3>フックが失敗したとき</h3>
- <p>フックの持ち時間は、リクエスト、プロバイダへの 1 回の試行、または応答全体で 250 ms、1 イベントで 50 ms、読み込みで 1 秒です。例外を投げたフック、時間切れになったフック、想定外の値を返したフックは、なかったものとして扱われます。つまりリクエストやイベントは元のまま送られます。設定 → プラグインには、ミドルウェアごとのフック、呼び出し回数、1 回あたりの平均時間、失敗回数と直近のエラーが表示されます。フックが見るのは、エージェントが送ったものと受け取るものそのものです。送信時は magpie のマスキングより前、受信時はマスク解除の後です。magpie 自身のリクエスト(スレッドのタイトル、ルーターの分類器)はミドルウェアを通りません。</p>
+ <p>フックの持ち時間は、リクエスト、プロバイダへの 1 回の試行、または応答全体で 250 ms、1 イベントで 50 ms、読み込みで 1 秒です。例外を投げたフック、時間切れになったフック、想定外の値を返したフックは、なかったものとして扱われます。つまりリクエストやイベントは元のまま送られます。設定 → プラグインには、ミドルウェアごとのフック、呼び出し回数、1 回あたりの平均時間、失敗回数と直近のエラーが表示されます。フックが見るのは、エージェントが送ったものと受け取るものそのものです。送信時は magpie のマスキングより前、受信時はマスク解除の後です。magpie 自身のリクエスト(スレッドのタイトル、ルーターの分類器、エージェントのリクエストのために行うウェブ検索と画像の説明)はミドルウェアを通りません。</p>
<h3 id="gateway-points"><a class="anchor" href="#gateway-points">ルーティングと使用量の行</a></h3>
<p>同じモジュールは <code>strategies</code> と <code>onUsage</code> もエクスポートできます。そのファイルは <code>*.gateway.js</code> と名付けてもよく、パッケージなら <code>"magpie": { "gateway": "./gw.js" }</code> と書けます。</p>
diff --git a/site/public/docs/plugins.html b/site/public/docs/plugins.html
index 7a06fa332..a28f3a2c9 100644
--- a/site/public/docs/plugins.html
+++ b/site/public/docs/plugins.html
@@ -710,7 +710,7 @@ magpie plugin options model-map '{"mapping": {"fast": "deepseek/deepseek-chat"}}
<p><code>export const events = ["content_block_delta"]</code> calls <code>onEvent</code> only for events of those names (Anthropic Messages and Responses name theirs). Chat Completions and Gemini events have no name, so leave <code>events</code> out for them. An event holds a piece of the reply, so a word can be split over two events. A <code>data: [DONE]</code> line and comments pass by unseen. A compressed reply passes as it is.</p>
<p><code>ctx</code> has <code>protocol</code> (<code>anthropic</code>, <code>chat</code>, <code>responses</code> or <code>gemini</code>), <code>model</code> and <code>stream</code> as the agent asked, <code>path</code>, <code>agent</code> when magpie knows it, <code>options</code> (the plugin’s <a href="#middleware-options">options</a>) and <code>state</code>, an object of the request’s own.</p>
<h3>When a hook fails</h3>
- <p>A hook has 250 ms for a request, a provider’s try or a whole reply and 50 ms for an event; loading has a second. A hook that throws, runs out of time or returns something else goes on as if it weren’t there: the request or the event is sent as it was. Settings → Plugins shows each middleware’s hooks, how many calls it had, the average time each took and how many failed, with the last error. The hooks see what the agent sent and gets back, before magpie’s redaction on the way out and after it on the way back. magpie’s own requests (thread titles, the router’s classifier) don’t go through middleware.</p>
+ <p>A hook has 250 ms for a request, a provider’s try or a whole reply and 50 ms for an event; loading has a second. A hook that throws, runs out of time or returns something else goes on as if it weren’t there: the request or the event is sent as it was. Settings → Plugins shows each middleware’s hooks, how many calls it had, the average time each took and how many failed, with the last error. The hooks see what the agent sent and gets back, before magpie’s redaction on the way out and after it on the way back. magpie’s own requests (thread titles, the router’s classifier, the web searches and image descriptions it makes for an agent’s request) don’t go through middleware.</p>
<h3 id="gateway-points"><a class="anchor" href="#gateway-points">Routings and usage rows</a></h3>
<p>The same module can also export <code>strategies</code> and <code>onUsage</code>. Such a file can be named <code>*.gateway.js</code>, and a package can say <code>"magpie": { "gateway": "./gw.js" }</code>.</p>
diff --git a/site/public/docs/zh/plugins.html b/site/public/docs/zh/plugins.html
index db1fa415b..c73659dc4 100644
--- a/site/public/docs/zh/plugins.html
+++ b/site/public/docs/zh/plugins.html
@@ -710,7 +710,7 @@ magpie plugin options model-map '{"mapping": {"fast": "deepseek/deepseek-chat"}}
<p><code>export const events = ["content_block_delta"]</code> 让 <code>onEvent</code> 只处理这些名字的事件(Anthropic Messages 和 Responses 的事件有名字)。Chat Completions 和 Gemini 的事件没有名字,处理它们时不要写 <code>events</code>。一个事件只是回复的一小段,一个词可能被拆在两个事件里。<code>data: [DONE]</code> 行和注释直接放行,不经过钩子。压缩过的回复原样放行。</p>
<p><code>ctx</code> 里有 <code>protocol</code>(<code>anthropic</code>、<code>chat</code>、<code>responses</code> 或 <code>gemini</code>)、agent 请求的 <code>model</code> 和 <code>stream</code>、<code>path</code>、magpie 知道时的 <code>agent</code>、<code>options</code>(这个插件的<a href="#middleware-options">选项</a>),以及 <code>state</code>:每个请求独有的对象。</p>
<h3>钩子出错时</h3>
- <p>处理请求、一次供应商尝试或完整回复时,钩子有 250 毫秒;处理一个事件时有 50 毫秒;加载有 1 秒。钩子抛出异常、超时或返回了别的东西,都当作它不存在:请求或事件按原样发送。设置 → 插件会显示每个中间件的钩子、调用次数、平均每次耗时和失败次数,以及最近一次错误。钩子看到的是 agent 发出和收到的原始内容:发出时在 magpie 脱敏之前,返回时在还原之后。magpie 自己的请求(会话标题、路由分类器)不经过中间件。</p>
+ <p>处理请求、一次供应商尝试或完整回复时,钩子有 250 毫秒;处理一个事件时有 50 毫秒;加载有 1 秒。钩子抛出异常、超时或返回了别的东西,都当作它不存在:请求或事件按原样发送。设置 → 插件会显示每个中间件的钩子、调用次数、平均每次耗时和失败次数,以及最近一次错误。钩子看到的是 agent 发出和收到的原始内容:发出时在 magpie 脱敏之前,返回时在还原之后。magpie 自己的请求(会话标题、路由分类器,以及它为 agent 的请求做的网页搜索和图片描述)不经过中间件。</p>
<h3 id="gateway-points"><a class="anchor" href="#gateway-points">路由与用量记录</a></h3>
<p>同一个模块还可以导出 <code>strategies</code> 和 <code>onUsage</code>。这样的文件可以命名为 <code>*.gateway.js</code>,包里也可以写 <code>"magpie": { "gateway": "./gw.js" }</code>。</p>