Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

gateway: embeddings and speech reach a plugin through its fetch whatever API its chat is on

yetone6cb73ec4f0a7
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index eca88ad5d..756c2ff58 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -471,7 +471,8 @@ before its route began (`TestDecisionTriesShareTheClassifierWait`). A gateway ke
(`allowedCandidates`), and refused 403 when none of them serves the model.
`TestDecisionProviderKeysFailOver`.
[`embeddings.go`](../../internal/gateway/embeddings.go) relays retrieval
-bodies and options (`serveRetrieval`), and
+bodies and options (`serveRetrieval`, under `Provider.OpenAIBase`: a
+plugin's fetch whatever API its chat models are on), and
[`transcribe.go`](../../internal/gateway/transcribe.go) speech-to-text's
multipart forms through it as sent, rewriting only the model and a masked
prompt under the form's own boundary (#2053), and
diff --git a/docs/subsystems/provider-plugins.md b/docs/subsystems/provider-plugins.md
index 99696b949..c7ca9fe83 100644
--- a/docs/subsystems/provider-plugins.md
+++ b/docs/subsystems/provider-plugins.md
@@ -84,6 +84,8 @@ A plugin in front of an OpenAI-compatible API draws with the image models its li
A plugin's model on Gemini's API (`@ai-sdk/google`, so `Provider.APIs` is Code Assist alone) draws as a Google sign-in's image models do (XYenon's Google AI Studio plugin, XYenon/magpie-google-ai-studio). `Drawers` lists it when its id names an image model (`catalog.DrawsID`) or the plugin says it gives images (`capabilities.output.image`, or models.dev's `modalities.output`, read into `OutputModalities`), and it isn't an Imagen (`catalog.ImagesAPI`, which draws on `:predict`, not served). `draw` sends it to `drawCodeAssist` (`pluginDrawsGemini`): one `generateContent` per image through the plugin's `fetch`, with `responseModalities` `["TEXT","IMAGE"]` (`Request.draws` asks for it of a model not named for images), and an edit's pictures as the prompt's `inlineData`; the pictures come back from the answer's `inlineData`. The images routes' key holds and usage apply as for any drawer. `TestGeminiPluginDraws`.
+Embeddings, rerank, speech and transcriptions reach a plugin provider whatever API its chat models are spoken on: `serveRetrieval` sends them under `Provider.OpenAIBase`, which is `plugin://<id>/v1` for a plugin (its Chat base for any other provider), so they arrive OpenAI-shaped at the plugin's `fetch`, at `<base>/embeddings`, `/rerank`, `/audio/speech` or `/audio/transcriptions` under the base `pluginFetch` picks for the body's model, with the model's own id. The plugin answers them or puts them in its vendor's shape (XYenon's Google AI Studio plugin: Gemini's `embedContent`, a TTS model's `generateContent`); magpie converts none of them to Gemini's API. Before, a plugin with no `@ai-sdk/openai-compatible` model was turned away 400 "has no OpenAI-style API". The model is named `provider/model`; the retrieval routes' key holds, account holds, masking and usage apply. `TestGooglePluginTakesEmbeddingsAndSpeech`.
+
A plugin can instead mark single models of its list with `searches: true` (on the `config` hook's model or the `models` hook's list): a model that searches the web by itself when asked in plain words, such as Trae's search model. The host passes it through (`plugin.Model.Searches`), `pluginCatalog` makes it the catalog model's `WebSearch` "native", and the gateway offers the provider in Settings › Web search's picker and its own pick with those models alone (`pluginSearches`, `partSearcher`).
## A plugin's service tiers
diff --git a/internal/gateway/embeddings.go b/internal/gateway/embeddings.go
index 314d04414..ddf552063 100644
--- a/internal/gateway/embeddings.go
+++ b/internal/gateway/embeddings.go
@@ -212,7 +212,7 @@ func (s *Server) serveRetrieval(w http.ResponseWriter, r *http.Request, path, op
for i, c := range tries {
last := i == len(tries)-1
call.Provider, went = c.p.ID, c.model
- base := strings.TrimRight(c.p.Base(provider.Chat), "/")
+ base := strings.TrimRight(c.p.OpenAIBase(), "/")
tried := side.try(c)
if base == "" {
msg := fmt.Sprintf("%s has no OpenAI-style API for %s", c.p.Name, path)
diff --git a/internal/gateway/plugin_retrieval_test.go b/internal/gateway/plugin_retrieval_test.go
new file mode 100644
index 000000000..c28c65f32
--- /dev/null
+++ b/internal/gateway/plugin_retrieval_test.go
@@ -0,0 +1,89 @@
+package gateway
+
+import (
+ "fmt"
+ "io"
+ "net/http"
+ "strings"
+ "sync"
+ "testing"
+
+ "github.com/yetone/magpie/internal/access"
+ "github.com/yetone/magpie/internal/provider"
+)
+
+// A plugin whose models are all on Gemini's API (XYenon's Google AI Studio
+// plugin) still takes embeddings and speech: they go OpenAI-shaped to its
+// fetch, at <its base>/embeddings and /audio/speech, for the plugin to
+// put in its vendor's shape, with the model's own id. The old gateway
+// turned them away 400, the provider having no chat base. A gateway key
+// held to other models is refused, and each call is in the usage ledger.
+func TestGooglePluginTakesEmbeddingsAndSpeech(t *testing.T) {
+ t.Setenv("FAKE_GOOGLE", "1")
+ t.Setenv("FAKE_KINDS", "1")
+ var mu sync.Mutex
+ sent := map[string][]string{}
+ pid := besideFake(t, "google-ai-studio", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ b, _ := io.ReadAll(r.Body)
+ mu.Lock()
+ sent[r.URL.Path] = append(sent[r.URL.Path], string(b))
+ mu.Unlock()
+ switch r.URL.Path {
+ case "/v1/embeddings":
+ w.Header().Set("Content-Type", "application/json")
+ fmt.Fprint(w, `{"object":"list","data":[{"object":"embedding","index":0,"embedding":[0.1,-0.2]}],"model":"gemini-embedding-001","usage":{"prompt_tokens":3,"total_tokens":3}}`)
+ case "/v1/audio/speech":
+ w.Header().Set("Content-Type", "audio/wav")
+ io.WriteString(w, "RIFF-fake-wav")
+ default:
+ w.WriteHeader(404)
+ }
+ }))
+ p, err := provider.Find(pid)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if p.Base(provider.Chat) != "" {
+ t.Fatalf("the provider has a chat base %q: every model of it should be on Gemini's API", p.Base(provider.Chat))
+ }
+ s := New()
+ code, raw := postRetrieval(t, s, "/v1/embeddings", `{"model":"`+pid+`/gemini-embedding-001","input":"hello"}`)
+ if code != 200 || !strings.Contains(raw, `"embedding":[0.1,-0.2]`) {
+ t.Fatalf("embeddings: %d %s", code, raw)
+ }
+ code, _, audio := postSpeech(t, s, "/v1/audio/speech", `{"model":"`+pid+`/gemini-2.5-flash-preview-tts","input":"Hello from the magpie.","voice":"Kore"}`)
+ if code != 200 || string(audio) != "RIFF-fake-wav" {
+ t.Fatalf("speech: %d %s", code, audio)
+ }
+ mu.Lock()
+ if e := sent["/v1/embeddings"]; len(e) != 1 || !strings.Contains(e[0], `"model":"gemini-embedding-001"`) || !strings.Contains(e[0], `"input":"hello"`) {
+ t.Fatalf("embeddings were sent %v", e)
+ }
+ if a := sent["/v1/audio/speech"]; len(a) != 1 || !strings.Contains(a[0], `"model":"gemini-2.5-flash-preview-tts"`) || !strings.Contains(a[0], `"voice":"Kore"`) {
+ t.Fatalf("speech was sent %v", a)
+ }
+ mu.Unlock()
+ if rows := usageRows(t, "embeddings", 1); len(rows) != 1 || rows[0].Provider != pid || rows[0].Model != "gemini-embedding-001" || rows[0].Input != 3 || rows[0].Status != 200 {
+ t.Fatalf("embeddings usage %+v", rows)
+ }
+ if rows := usageRows(t, "speech", 1); len(rows) != 1 || rows[0].Provider != pid || rows[0].Status != 200 {
+ t.Fatalf("speech usage %+v", rows)
+ }
+
+ keys, secrets := newCaller(t, "Elsewhere")
+ if _, err := access.Update("models-key", access.Change{Key: keys[0].ID, Models: []string{"elsewhere/*"}}); err != nil {
+ t.Fatal(err)
+ }
+ h := s.Handler()
+ if code, _, raw := asKey(t, h, secrets[0], "POST", "/v1/embeddings", `{"model":"`+pid+`/gemini-embedding-001","input":"hello"}`); code != 403 {
+ t.Fatalf("a key held to other models: %d %s", code, raw)
+ }
+ if code, _, raw := asKey(t, h, secrets[0], "POST", "/v1/audio/speech", `{"model":"`+pid+`/gemini-2.5-flash-preview-tts","input":"hi"}`); code != 403 {
+ t.Fatalf("a key held to other models, speech: %d %s", code, raw)
+ }
+ mu.Lock()
+ defer mu.Unlock()
+ if len(sent["/v1/embeddings"]) != 1 || len(sent["/v1/audio/speech"]) != 1 {
+ t.Fatalf("a refused key's request reached the plugin: %v", sent)
+ }
+}
diff --git a/internal/plugin/testdata/fake/index.js b/internal/plugin/testdata/fake/index.js
index d0b3b9479..e8b487879 100644
--- a/internal/plugin/testdata/fake/index.js
+++ b/internal/plugin/testdata/fake/index.js
@@ -93,6 +93,10 @@ export const FakePlugin = async ({ client }) => ({
cfg.provider[ID].models["sensenova-u1.5-lite"] = { name: "sensenova-u1.5-lite", limit: { context: 131072, output: 32768 } }
cfg.provider[ID].models["flux-2"] = { name: "Flux 2", provider: { npm: "@ai-sdk/anthropic" }, limit: { context: 1000, output: 100 } }
}
+ // $FAKE_GOOGLE: the provider's models are on Gemini's API, as XYenon's
+ // Google AI Studio plugin gives them all (@ai-sdk/google), so it has
+ // no OpenAI-compatible model
+ if (process.env.FAKE_GOOGLE) cfg.provider[ID].npm = "@ai-sdk/google"
// $FAKE_GEMINI_DRAWS: image models on Gemini's API, as XYenon's Google
// AI Studio plugin lists them: one named for images, one that only
// says it gives them, and an Imagen, which draws on :predict instead
diff --git a/internal/provider/plugins.go b/internal/provider/plugins.go
index b03333fc8..9f66a2190 100644
--- a/internal/provider/plugins.go
+++ b/internal/provider/plugins.go
@@ -123,6 +123,18 @@ func (p Provider) PluginProvider() string {
return p.Account.plugin.ID
}
+// OpenAIBase is where p takes the OpenAI-shaped requests that are no chat
+// (embeddings, rerank, speech, transcriptions): its Chat base, and a
+// plugin's fetch under plugin://<id>/v1 whatever API its chat models are
+// spoken on, for the plugin to answer or put in its vendor's shape
+// (XYenon's Google AI Studio plugin: Gemini's embedContent, its TTS).
+func (p Provider) OpenAIBase() string {
+ if p.IsPlugin() {
+ return pluginBase + p.Account.plugin.ID + "/v1"
+ }
+ return p.Base(Chat)
+}
+
// PluginOpenAICompatible is whether p is a plugin's provider that speaks
// model on an OpenAI-compatible API (@ai-sdk/openai-compatible), whose
// other routes, the images API among them, its fetch sends under the same