Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

video: a plugin's model that gives video makes videos through its fetch on the account

yetonef11bf27c1f3f
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index 756c2ff58..f165c900e 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -500,7 +500,9 @@ same OpenAI way (#1446, `customVideomakers`), but only with the models its
own fetched list marks `"kind": "video"` (`catalog.LiveVideomakers`) or the
user marks a video model (settings' `ModelKinds`, `provider.SetModelKind`,
a model chip's right-click *Makes…* in the editor, #1399); a name alone
-never makes a model a video maker. A model marked an image model is drawn
+never makes a model a video maker. A plugin's provider makes videos with the
+models it says give video, through its fetch on the account that started
+each (`keyMark`, `markedKey`; see [provider plugins](provider-plugins.md)). A model marked an image model is drawn
with (`Drawers`) and leaves the video makers, and one marked a video model
leaves the drawers; only a provider set up with a base URL can be marked
(`MarksKinds`). Start, poll and content go to
diff --git a/docs/subsystems/provider-plugins.md b/docs/subsystems/provider-plugins.md
index c7ca9fe83..e3671640b 100644
--- a/docs/subsystems/provider-plugins.md
+++ b/docs/subsystems/provider-plugins.md
@@ -86,6 +86,8 @@ A plugin's model on Gemini's API (`@ai-sdk/google`, so `Provider.APIs` is Code A
Embeddings, rerank, speech and transcriptions reach a plugin provider whatever API its chat models are spoken on: `serveRetrieval` sends them under `Provider.OpenAIBase`, which is `plugin://<id>/v1` for a plugin (its Chat base for any other provider), so they arrive OpenAI-shaped at the plugin's `fetch`, at `<base>/embeddings`, `/rerank`, `/audio/speech` or `/audio/transcriptions` under the base `pluginFetch` picks for the body's model, with the model's own id. The plugin answers them or puts them in its vendor's shape (XYenon's Google AI Studio plugin: Gemini's `embedContent`, a TTS model's `generateContent`); magpie converts none of them to Gemini's API. Before, a plugin with no `@ai-sdk/openai-compatible` model was turned away 400 "has no OpenAI-style API". The model is named `provider/model`; the retrieval routes' key holds, account holds, masking and usage apply. `TestGooglePluginTakesEmbeddingsAndSpeech`.
+A plugin's model that says it gives video (`capabilities.output.video`, or `modalities.output: ["video"]`) makes videos (`customVideomakers`; the Grok plugin keeps its own). Start, poll and content go OpenAI-shaped through the plugin's `fetch` on the account, to `<base>/videos`, `/videos/{id}` and `/videos/{id}/content` (`videoBase`, `Provider.OpenAIBase`): a POST of OpenAI's video body (`remoteVideoBody`: `model` as the plugin's id for it, `prompt`, `seconds`, `size`, `input_reference` as a data URL), answered `{id, object: "video", status, progress, model}`; the id is the vendor's, path-escaped in the GET. The poll and content GETs carry no model, so `pluginFetch` sends them under the loader's base (else the provider's `api`, else its npm's). The video's id names the account that started it (`keyMark`: `a` + `signer`), and `markedKey` asks after it on that account of the plugin's (`AlsoOn`) while it is signed in. Key account and model holds apply to all three routes (`videoAsker`, `videoModelHeld`, which needs the answer's `model`), and the start is in the usage ledger. `TestPluginMakesVideos`.
+
A plugin can instead mark single models of its list with `searches: true` (on the `config` hook's model or the `models` hook's list): a model that searches the web by itself when asked in plain words, such as Trae's search model. The host passes it through (`plugin.Model.Searches`), `pluginCatalog` makes it the catalog model's `WebSearch` "native", and the gateway offers the provider in Settings › Web search's picker and its own pick with those models alone (`pluginSearches`, `partSearcher`).
## A plugin's service tiers
diff --git a/internal/gateway/video.go b/internal/gateway/video.go
index 05a5f2bd6..0cd0cb23c 100644
--- a/internal/gateway/video.go
+++ b/internal/gateway/video.go
@@ -99,8 +99,20 @@ func Videomakers(p provider.Provider) []catalog.Model {
// videos (provider.SetModelKind, #1399). They are asked at the
// OpenAI-shaped videos API under its base, as another magpie's are. A
// model's name alone makes none of them one: a list that doesn't say so,
-// and a user who doesn't, leave the provider making no videos.
+// and a user who doesn't, leave the provider making no videos. A plugin's
+// are the models it says give video (capabilities.output.video), asked
+// through its fetch on the account (XYenon's Google AI Studio plugin, Veo).
func customVideomakers(p provider.Provider) []catalog.Model {
+ if p.IsPlugin() && !drawsGrok(p) {
+ var out []catalog.Model
+ for _, m := range p.Available() {
+ if slices.Contains(m.OutputModalities, "video") {
+ m.Provider = p.ID
+ out = append(out, m)
+ }
+ }
+ return out
+ }
if p.Account != nil || p.IsRemoteMagpie() || len(p.PlanVideos()) > 0 || filmsMiniMax(p) || videoBase(p) == "" {
return nil
}
@@ -120,9 +132,10 @@ func customVideomakers(p provider.Provider) []catalog.Model {
// videoBase is the root of the OpenAI-shaped videos API of another magpie
// or a provider set up with a base URL: its Chat base, or a Responses-only
-// relay's Responses base, under which it serves the images API too.
+// relay's Responses base, under which it serves the images API too; a
+// plugin's fetch whatever API its chat is on (Provider.OpenAIBase).
func videoBase(p provider.Provider) string {
- return strings.TrimRight(cmp.Or(p.Base(provider.Chat), p.Base(provider.Responses)), "/")
+ return strings.TrimRight(cmp.Or(p.OpenAIBase(), p.Base(provider.Responses)), "/")
}
// filmsCustom is whether p makes model at its OpenAI-shaped videos API, as
@@ -134,8 +147,12 @@ func filmsCustom(p provider.Provider, model string) bool {
// keyMark is the mark of the key p sends with, in the id of a video it
// started at its OpenAI-shaped videos API: the vendor keeps a video for the
// key that started it, so it is asked after on that key, whichever comes
-// first now. "0" for a provider with no key (a local server).
+// first now. "0" for a provider with no key (a local server); a plugin's
+// account's signer, the vendor keeping it for that account.
func keyMark(p provider.Provider) string {
+ if p.IsPlugin() {
+ return "a" + signer(p)
+ }
if p.Key == "" {
return "0"
}
@@ -151,6 +168,15 @@ func customVideoID(p provider.Provider, theirs string, t time.Time) string {
// markedKey is p on its key whose mark is by (keyMark), while that key is
// still on.
func markedKey(p provider.Provider, by string) (provider.Provider, bool) {
+ if p.IsPlugin() {
+ // the plugin's account that started it, while it is still on
+ for _, q := range append([]provider.Provider{p}, p.AlsoOn()...) {
+ if keyMark(q) == by {
+ return q, true
+ }
+ }
+ return p, false
+ }
if p.Key == "" {
return p, by == "0"
}
diff --git a/internal/gateway/video_plugin_test.go b/internal/gateway/video_plugin_test.go
new file mode 100644
index 000000000..ae5a62296
--- /dev/null
+++ b/internal/gateway/video_plugin_test.go
@@ -0,0 +1,122 @@
+package gateway
+
+import (
+ "context"
+ "io"
+ "net/http"
+ "slices"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/yetone/magpie/internal/access"
+ "github.com/yetone/magpie/internal/plugin"
+ "github.com/yetone/magpie/internal/provider"
+)
+
+// A plugin's model that says it gives video (Veo, on XYenon's Google AI
+// Studio plugin) makes videos: started, polled and fetched at the
+// OpenAI-shaped videos API under the plugin's base, through its fetch, on
+// the account that started it, whichever is first now. A gateway key held
+// to another account may not ask after it, nor one held to other models
+// start or ask after one; the start is in the usage ledger. The old
+// gateway listed no plugin's video models and turned the start away 400.
+func TestPluginMakesVideos(t *testing.T) {
+ t.Setenv("FAKE_GOOGLE", "1")
+ t.Setenv("FAKE_VIDEO", "1")
+ var mu sync.Mutex
+ var asked []string
+ const vid = "models/veo-3.1-generate-preview/operations/abc.1"
+ pid := besideFake(t, "google-ai-studio", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ b, _ := io.ReadAll(r.Body)
+ key := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")
+ mu.Lock()
+ asked = append(asked, r.Method+" "+r.URL.EscapedPath()+" "+key)
+ mu.Unlock()
+ w.Header().Set("Content-Type", "application/json")
+ switch {
+ case r.Method == "POST" && r.URL.Path == "/v1/videos":
+ if !strings.Contains(string(b), `"model":"veo-3.1-generate-preview"`) || !strings.Contains(string(b), `"prompt":"a magpie"`) {
+ w.WriteHeader(400)
+ io.WriteString(w, `{"error":{"message":"sent `+strings.ReplaceAll(string(b), `"`, `'`)+`"}}`)
+ return
+ }
+ io.WriteString(w, `{"id":"`+vid+`","object":"video","status":"queued","progress":0,"model":"veo-3.1-generate-preview"}`)
+ case r.Method == "GET" && r.URL.Path == "/v1/videos/"+vid:
+ io.WriteString(w, `{"id":"`+vid+`","object":"video","status":"completed","progress":100,"model":"veo-3.1-generate-preview"}`)
+ case r.Method == "GET" && r.URL.Path == "/v1/videos/"+vid+"/content":
+ w.Header().Set("Content-Type", "video/mp4")
+ io.WriteString(w, "\x00\x00\x00\x18ftypmp42fake-veo")
+ default:
+ w.WriteHeader(404)
+ }
+ }))
+ ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
+ defer cancel()
+ if _, err := plugin.APIKey(ctx, "google-ai-studio", 0, nil, "k2", plugin.NewAccount); err != nil {
+ t.Fatal(err)
+ }
+ p, err := provider.Find(pid)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if ms := Videomakers(*p); len(ms) != 1 || ms[0].ID != "veo-3.1-generate-preview" || ms[0].Provider != pid {
+ t.Fatalf("videomakers %v, want veo-3.1-generate-preview alone", ms)
+ }
+ accts := append([]provider.Provider{*p}, p.AlsoOn()...)
+ if len(accts) != 2 {
+ t.Fatalf("%d accounts, want 2", len(accts))
+ }
+ onSecond := "" // the second account's id, as a key's hold names it
+ for _, q := range accts[1:] {
+ onSecond = pid + "/" + q.AccountID()
+ }
+ keys, secrets := newCaller(t, "OnSecond", "OnFirst", "OtherModels")
+ for i, c := range []access.Change{
+ {Key: keys[0].ID, Accounts: []string{onSecond}},
+ {Key: keys[1].ID, Accounts: []string{pid + "/" + accts[0].AccountID()}},
+ {Key: keys[2].ID, Models: []string{"elsewhere/*"}},
+ } {
+ if _, err := access.Update([]string{"accounts-key", "accounts-key", "models-key"}[i], c); err != nil {
+ t.Fatal(err)
+ }
+ }
+ h := New().Handler()
+ code, obj, raw := asKey(t, h, secrets[0], "POST", "/v1/videos", `{"model":"`+pid+`/veo-3.1-generate-preview","prompt":"a magpie"}`)
+ id, _ := obj["id"].(string)
+ if code != 200 || id == "" {
+ t.Fatalf("start: %d %s", code, raw)
+ }
+ // a caller with no hold asks after it on the account that started it
+ if code, obj, raw = asKey(t, h, "", "GET", "/v1/videos/"+id, ""); code != 200 || obj["status"] != "completed" || obj["model"] != pid+"/veo-3.1-generate-preview" {
+ t.Fatalf("poll: %d %s", code, raw)
+ }
+ if code, _, raw = asKey(t, h, secrets[0], "GET", "/v1/videos/"+id+"/content", ""); code != 200 || !strings.HasSuffix(string(raw), "fake-veo") {
+ t.Fatalf("content: %d %s", code, raw)
+ }
+ mu.Lock()
+ got := slices.Clone(asked)
+ mu.Unlock()
+ // the content is asked for after the model making it (videoModelHeld)
+ if len(got) != 4 || !strings.HasPrefix(got[0], "POST /v1/videos ") || !strings.HasSuffix(got[3], "/content k2") {
+ t.Fatalf("the vendor was asked %v", got)
+ }
+ for _, a := range got {
+ if !strings.HasSuffix(a, " k2") {
+ t.Fatalf("asked on another account than the second: %v", got)
+ }
+ }
+ if code, _, raw = asKey(t, h, secrets[1], "GET", "/v1/videos/"+id, ""); code != 403 {
+ t.Fatalf("a key held to the first account asked after the second's video: %d %s", code, raw)
+ }
+ if code, _, raw = asKey(t, h, secrets[2], "GET", "/v1/videos/"+id, ""); code != 403 {
+ t.Fatalf("a key held to other models asked after it: %d %s", code, raw)
+ }
+ if code, _, raw = asKey(t, h, secrets[2], "POST", "/v1/videos", `{"model":"`+pid+`/veo-3.1-generate-preview","prompt":"a magpie"}`); code != 403 {
+ t.Fatalf("a key held to other models started one: %d %s", code, raw)
+ }
+ if rows := usageRows(t, "generate_content", 1); len(rows) != 1 || rows[0].Provider != pid || rows[0].Model != "veo-3.1-generate-preview" || rows[0].Status != 200 {
+ t.Fatalf("usage %+v", rows)
+ }
+}
diff --git a/internal/plugin/testdata/fake/index.js b/internal/plugin/testdata/fake/index.js
index e8b487879..efd3e27a9 100644
--- a/internal/plugin/testdata/fake/index.js
+++ b/internal/plugin/testdata/fake/index.js
@@ -97,6 +97,9 @@ export const FakePlugin = async ({ client }) => ({
// Google AI Studio plugin gives them all (@ai-sdk/google), so it has
// no OpenAI-compatible model
if (process.env.FAKE_GOOGLE) cfg.provider[ID].npm = "@ai-sdk/google"
+ // $FAKE_VIDEO: a video model, as a Gemini plugin lists Veo: it says it
+ // gives video, nothing else
+ if (process.env.FAKE_VIDEO) cfg.provider[ID].models["veo-3.1-generate-preview"] = { name: "Veo 3.1", modalities: { input: ["text", "image"], output: ["video"] }, limit: { context: 1000, output: 100 } }
// $FAKE_GEMINI_DRAWS: image models on Gemini's API, as XYenon's Google
// AI Studio plugin lists them: one named for images, one that only
// says it gives them, and an Imagen, which draws on :predict instead