Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
video: a plugin's model that gives video makes videos through its fetch on the account
yetonef11bf27c1f3f
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex 756c2ff58..f165c900e 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -500,7 +500,9 @@ same OpenAI way (#1446, `customVideomakers`), but only with the models itsown fetched list marks `"kind": "video"` (`catalog.LiveVideomakers`) or theuser marks a video model (settings' `ModelKinds`, `provider.SetModelKind`,a model chip's right-click *Makes…* in the editor, #1399); a name alone-never makes a model a video maker. A model marked an image model is drawn+never makes a model a video maker. A plugin's provider makes videos with the+models it says give video, through its fetch on the account that started+each (`keyMark`, `markedKey`; see [provider plugins](provider-plugins.md)). A model marked an image model is drawnwith (`Drawers`) and leaves the video makers, and one marked a video modelleaves the drawers; only a provider set up with a base URL can be marked(`MarksKinds`). Start, poll and content go todiff --git a/docs/subsystems/provider-plugins.md b/docs/subsystems/provider-plugins.mdindex c7ca9fe83..e3671640b 100644--- a/docs/subsystems/provider-plugins.md+++ b/docs/subsystems/provider-plugins.md@@ -86,6 +86,8 @@ A plugin's model on Gemini's API (`@ai-sdk/google`, so `Provider.APIs` is Code AEmbeddings, rerank, speech and transcriptions reach a plugin provider whatever API its chat models are spoken on: `serveRetrieval` sends them under `Provider.OpenAIBase`, which is `plugin://<id>/v1` for a plugin (its Chat base for any other provider), so they arrive OpenAI-shaped at the plugin's `fetch`, at `<base>/embeddings`, `/rerank`, `/audio/speech` or `/audio/transcriptions` under the base `pluginFetch` picks for the body's model, with the model's own id. The plugin answers them or puts them in its vendor's shape (XYenon's Google AI Studio plugin: Gemini's `embedContent`, a TTS model's `generateContent`); magpie converts none of them to Gemini's API. Before, a plugin with no `@ai-sdk/openai-compatible` model was turned away 400 "has no OpenAI-style API". The model is named `provider/model`; the retrieval routes' key holds, account holds, masking and usage apply. `TestGooglePluginTakesEmbeddingsAndSpeech`.+A plugin's model that says it gives video (`capabilities.output.video`, or `modalities.output: ["video"]`) makes videos (`customVideomakers`; the Grok plugin keeps its own). Start, poll and content go OpenAI-shaped through the plugin's `fetch` on the account, to `<base>/videos`, `/videos/{id}` and `/videos/{id}/content` (`videoBase`, `Provider.OpenAIBase`): a POST of OpenAI's video body (`remoteVideoBody`: `model` as the plugin's id for it, `prompt`, `seconds`, `size`, `input_reference` as a data URL), answered `{id, object: "video", status, progress, model}`; the id is the vendor's, path-escaped in the GET. The poll and content GETs carry no model, so `pluginFetch` sends them under the loader's base (else the provider's `api`, else its npm's). The video's id names the account that started it (`keyMark`: `a` + `signer`), and `markedKey` asks after it on that account of the plugin's (`AlsoOn`) while it is signed in. Key account and model holds apply to all three routes (`videoAsker`, `videoModelHeld`, which needs the answer's `model`), and the start is in the usage ledger. `TestPluginMakesVideos`.+A plugin can instead mark single models of its list with `searches: true` (on the `config` hook's model or the `models` hook's list): a model that searches the web by itself when asked in plain words, such as Trae's search model. The host passes it through (`plugin.Model.Searches`), `pluginCatalog` makes it the catalog model's `WebSearch` "native", and the gateway offers the provider in Settings › Web search's picker and its own pick with those models alone (`pluginSearches`, `partSearcher`).## A plugin's service tiersdiff --git a/internal/gateway/video.go b/internal/gateway/video.goindex 05a5f2bd6..0cd0cb23c 100644--- a/internal/gateway/video.go+++ b/internal/gateway/video.go@@ -99,8 +99,20 @@ func Videomakers(p provider.Provider) []catalog.Model {// videos (provider.SetModelKind, #1399). They are asked at the// OpenAI-shaped videos API under its base, as another magpie's are. A// model's name alone makes none of them one: a list that doesn't say so,-// and a user who doesn't, leave the provider making no videos.+// and a user who doesn't, leave the provider making no videos. A plugin's+// are the models it says give video (capabilities.output.video), asked+// through its fetch on the account (XYenon's Google AI Studio plugin, Veo).func customVideomakers(p provider.Provider) []catalog.Model {+ if p.IsPlugin() && !drawsGrok(p) {+ var out []catalog.Model+ for _, m := range p.Available() {+ if slices.Contains(m.OutputModalities, "video") {+ m.Provider = p.ID+ out = append(out, m)+ }+ }+ return out+ }if p.Account != nil || p.IsRemoteMagpie() || len(p.PlanVideos()) > 0 || filmsMiniMax(p) || videoBase(p) == "" {return nil}@@ -120,9 +132,10 @@ func customVideomakers(p provider.Provider) []catalog.Model {// videoBase is the root of the OpenAI-shaped videos API of another magpie// or a provider set up with a base URL: its Chat base, or a Responses-only-// relay's Responses base, under which it serves the images API too.+// relay's Responses base, under which it serves the images API too; a+// plugin's fetch whatever API its chat is on (Provider.OpenAIBase).func videoBase(p provider.Provider) string {- return strings.TrimRight(cmp.Or(p.Base(provider.Chat), p.Base(provider.Responses)), "/")+ return strings.TrimRight(cmp.Or(p.OpenAIBase(), p.Base(provider.Responses)), "/")}// filmsCustom is whether p makes model at its OpenAI-shaped videos API, as@@ -134,8 +147,12 @@ func filmsCustom(p provider.Provider, model string) bool {// keyMark is the mark of the key p sends with, in the id of a video it// started at its OpenAI-shaped videos API: the vendor keeps a video for the// key that started it, so it is asked after on that key, whichever comes-// first now. "0" for a provider with no key (a local server).+// first now. "0" for a provider with no key (a local server); a plugin's+// account's signer, the vendor keeping it for that account.func keyMark(p provider.Provider) string {+ if p.IsPlugin() {+ return "a" + signer(p)+ }if p.Key == "" {return "0"}@@ -151,6 +168,15 @@ func customVideoID(p provider.Provider, theirs string, t time.Time) string {// markedKey is p on its key whose mark is by (keyMark), while that key is// still on.func markedKey(p provider.Provider, by string) (provider.Provider, bool) {+ if p.IsPlugin() {+ // the plugin's account that started it, while it is still on+ for _, q := range append([]provider.Provider{p}, p.AlsoOn()...) {+ if keyMark(q) == by {+ return q, true+ }+ }+ return p, false+ }if p.Key == "" {return p, by == "0"}diff --git a/internal/gateway/video_plugin_test.go b/internal/gateway/video_plugin_test.gonew file mode 100644index 000000000..ae5a62296--- /dev/null+++ b/internal/gateway/video_plugin_test.go@@ -0,0 +1,122 @@+package gateway++import (+ "context"+ "io"+ "net/http"+ "slices"+ "strings"+ "sync"+ "testing"+ "time"++ "github.com/yetone/magpie/internal/access"+ "github.com/yetone/magpie/internal/plugin"+ "github.com/yetone/magpie/internal/provider"+)++// A plugin's model that says it gives video (Veo, on XYenon's Google AI+// Studio plugin) makes videos: started, polled and fetched at the+// OpenAI-shaped videos API under the plugin's base, through its fetch, on+// the account that started it, whichever is first now. A gateway key held+// to another account may not ask after it, nor one held to other models+// start or ask after one; the start is in the usage ledger. The old+// gateway listed no plugin's video models and turned the start away 400.+func TestPluginMakesVideos(t *testing.T) {+ t.Setenv("FAKE_GOOGLE", "1")+ t.Setenv("FAKE_VIDEO", "1")+ var mu sync.Mutex+ var asked []string+ const vid = "models/veo-3.1-generate-preview/operations/abc.1"+ pid := besideFake(t, "google-ai-studio", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ b, _ := io.ReadAll(r.Body)+ key := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")+ mu.Lock()+ asked = append(asked, r.Method+" "+r.URL.EscapedPath()+" "+key)+ mu.Unlock()+ w.Header().Set("Content-Type", "application/json")+ switch {+ case r.Method == "POST" && r.URL.Path == "/v1/videos":+ if !strings.Contains(string(b), `"model":"veo-3.1-generate-preview"`) || !strings.Contains(string(b), `"prompt":"a magpie"`) {+ w.WriteHeader(400)+ io.WriteString(w, `{"error":{"message":"sent `+strings.ReplaceAll(string(b), `"`, `'`)+`"}}`)+ return+ }+ io.WriteString(w, `{"id":"`+vid+`","object":"video","status":"queued","progress":0,"model":"veo-3.1-generate-preview"}`)+ case r.Method == "GET" && r.URL.Path == "/v1/videos/"+vid:+ io.WriteString(w, `{"id":"`+vid+`","object":"video","status":"completed","progress":100,"model":"veo-3.1-generate-preview"}`)+ case r.Method == "GET" && r.URL.Path == "/v1/videos/"+vid+"/content":+ w.Header().Set("Content-Type", "video/mp4")+ io.WriteString(w, "\x00\x00\x00\x18ftypmp42fake-veo")+ default:+ w.WriteHeader(404)+ }+ }))+ ctx, cancel := context.WithTimeout(context.Background(), time.Minute)+ defer cancel()+ if _, err := plugin.APIKey(ctx, "google-ai-studio", 0, nil, "k2", plugin.NewAccount); err != nil {+ t.Fatal(err)+ }+ p, err := provider.Find(pid)+ if err != nil {+ t.Fatal(err)+ }+ if ms := Videomakers(*p); len(ms) != 1 || ms[0].ID != "veo-3.1-generate-preview" || ms[0].Provider != pid {+ t.Fatalf("videomakers %v, want veo-3.1-generate-preview alone", ms)+ }+ accts := append([]provider.Provider{*p}, p.AlsoOn()...)+ if len(accts) != 2 {+ t.Fatalf("%d accounts, want 2", len(accts))+ }+ onSecond := "" // the second account's id, as a key's hold names it+ for _, q := range accts[1:] {+ onSecond = pid + "/" + q.AccountID()+ }+ keys, secrets := newCaller(t, "OnSecond", "OnFirst", "OtherModels")+ for i, c := range []access.Change{+ {Key: keys[0].ID, Accounts: []string{onSecond}},+ {Key: keys[1].ID, Accounts: []string{pid + "/" + accts[0].AccountID()}},+ {Key: keys[2].ID, Models: []string{"elsewhere/*"}},+ } {+ if _, err := access.Update([]string{"accounts-key", "accounts-key", "models-key"}[i], c); err != nil {+ t.Fatal(err)+ }+ }+ h := New().Handler()+ code, obj, raw := asKey(t, h, secrets[0], "POST", "/v1/videos", `{"model":"`+pid+`/veo-3.1-generate-preview","prompt":"a magpie"}`)+ id, _ := obj["id"].(string)+ if code != 200 || id == "" {+ t.Fatalf("start: %d %s", code, raw)+ }+ // a caller with no hold asks after it on the account that started it+ if code, obj, raw = asKey(t, h, "", "GET", "/v1/videos/"+id, ""); code != 200 || obj["status"] != "completed" || obj["model"] != pid+"/veo-3.1-generate-preview" {+ t.Fatalf("poll: %d %s", code, raw)+ }+ if code, _, raw = asKey(t, h, secrets[0], "GET", "/v1/videos/"+id+"/content", ""); code != 200 || !strings.HasSuffix(string(raw), "fake-veo") {+ t.Fatalf("content: %d %s", code, raw)+ }+ mu.Lock()+ got := slices.Clone(asked)+ mu.Unlock()+ // the content is asked for after the model making it (videoModelHeld)+ if len(got) != 4 || !strings.HasPrefix(got[0], "POST /v1/videos ") || !strings.HasSuffix(got[3], "/content k2") {+ t.Fatalf("the vendor was asked %v", got)+ }+ for _, a := range got {+ if !strings.HasSuffix(a, " k2") {+ t.Fatalf("asked on another account than the second: %v", got)+ }+ }+ if code, _, raw = asKey(t, h, secrets[1], "GET", "/v1/videos/"+id, ""); code != 403 {+ t.Fatalf("a key held to the first account asked after the second's video: %d %s", code, raw)+ }+ if code, _, raw = asKey(t, h, secrets[2], "GET", "/v1/videos/"+id, ""); code != 403 {+ t.Fatalf("a key held to other models asked after it: %d %s", code, raw)+ }+ if code, _, raw = asKey(t, h, secrets[2], "POST", "/v1/videos", `{"model":"`+pid+`/veo-3.1-generate-preview","prompt":"a magpie"}`); code != 403 {+ t.Fatalf("a key held to other models started one: %d %s", code, raw)+ }+ if rows := usageRows(t, "generate_content", 1); len(rows) != 1 || rows[0].Provider != pid || rows[0].Model != "veo-3.1-generate-preview" || rows[0].Status != 200 {+ t.Fatalf("usage %+v", rows)+ }+}diff --git a/internal/plugin/testdata/fake/index.js b/internal/plugin/testdata/fake/index.jsindex e8b487879..efd3e27a9 100644--- a/internal/plugin/testdata/fake/index.js+++ b/internal/plugin/testdata/fake/index.js@@ -97,6 +97,9 @@ export const FakePlugin = async ({ client }) => ({// Google AI Studio plugin gives them all (@ai-sdk/google), so it has// no OpenAI-compatible modelif (process.env.FAKE_GOOGLE) cfg.provider[ID].npm = "@ai-sdk/google"+ // $FAKE_VIDEO: a video model, as a Gemini plugin lists Veo: it says it+ // gives video, nothing else+ if (process.env.FAKE_VIDEO) cfg.provider[ID].models["veo-3.1-generate-preview"] = { name: "Veo 3.1", modalities: { input: ["text", "image"], output: ["video"] }, limit: { context: 1000, output: 100 } }// $FAKE_GEMINI_DRAWS: image models on Gemini's API, as XYenon's Google// AI Studio plugin lists them: one named for images, one that only// says it gives them, and an Imagen, which draws on :predict instead