Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

gateway: image generation is an MCP server on the gateway, /mcp/magpie/image

yetone402577c15aec
diff --git a/docs/reference.md b/docs/reference.md
index 7c8ff54ce..36a529bf3 100644
--- a/docs/reference.md
+++ b/docs/reference.md
@@ -547,6 +547,15 @@ or the turn cannot be identified, it uses Settings → Images → Image generati
turn's provider. An explicit `provider/model` keeps its destination; an upstream
image error does not retry through the setting. Magpie Image MCP is not required.
+The same model is an MCP server on the gateway at `/mcp/magpie/image`
+(Streamable HTTP; tools `generate_image`, `generate_video`,
+`generation_result`), for agents that aren't set up from the Library, such
+as magpie in Docker in gateway mode. Add `http://<host>:3425/mcp/magpie/image`
+as an HTTP MCP server; from another computer send a gateway key as
+`Authorization: Bearer <key>`. An image comes back in the tool's answer and
+at a URL on the gateway kept for an hour (the same key reads it), a video at
+`/v1/videos/<id>/content`. With Image generation Off the tool says so.
+
WorkBuddy (China)'s daily check-in (签到) is pressed from the app's Usage card
(*Check in now*), `magpie accounts checkin`, or `c` on the TUI's Usage page,
which checks in every account signed in on this computer at once; its
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index d58387049..bc6fe1bc9 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -28,7 +28,7 @@ backfilled; see [Gateway Session Aggregation](gateway-sessions.md).
| --- | --- | --- |
| Address and caller | `DefaultAddr`. The local token `magpie` and the per-agent `magpie-<agent>` (`TokenFor`). `agentOf` tells which agent asked, first by token, then by User-Agent. Claude Science, whose User-Agent is the Anthropic SDK's, is told by its client's `X-Anthropic-Surface: operon` (`isClaudeScience`). It is listed every model as `claude-magpie-<number>` (`scienceModels`; its picker keeps only a `claude-` id whose name isn't a lower-case slug), and a Claude id magpie doesn't serve (its default before a pick, its haiku side tasks) goes to the model its last turn with tools was on, else the first it is listed (`scienceTurn`, kept in `claude-science.model`). | [`gateway.go`](../../internal/gateway/gateway.go) |
| TLS | `--tls-cert`/`--tls-key` (`MAGPIE_TLS_CERT`, `MAGPIE_TLS_KEY`) make `ListenAndServe`, `Relisten` (both through `Server.listen`) and `magpie web`'s page (`ServeTLS`) wrap the listener in `tlsListener`. A connection from another computer is `tls.Server` at once, so its plain HTTP gets net/http's 400. One from this computer (`fromThisComputer`: loopback, or a peer address equal to the local one) is sniffed on its own goroutine, 10s at most (`sniffWait`), and 0x16 makes it TLS, so `URL()`, the agents, MCP servers, `ServedBy` and the healthcheck keep `http://127.0.0.1`. `certFiles.get` reloads the pair when either file's mtime or size changes; a pair that doesn't load keeps the one served (logged once per change). `tlsConfig` refuses half a pair or files that don't load before binding. HTTP/1.1 only over TLS (hijack, Codex's websocket). `Scheme()` gives the scheme of `LANURLs`, a scheme-less `MAGPIE_PUBLIC_URL` and the page's network links. Host and Origin checks (`foreignPage`, `rebound`) don't depend on the scheme. | [`tls.go`](../../internal/gateway/tls.go), [`lan.go`](../../internal/gateway/lan.go) |
-| HTTP surface | `Handler` routes `/v1/chat/completions`, `/v1/responses`, `/v1/messages` (each also without `/v1`), `/v1/models`, Gemini's `/v1beta/models/…`, the Codex backend path, `/v1/codex/models` (Codex's catalog shape for a Codex elsewhere with a gateway key, `codexCatalog`, kept to the key's models by `keyAllowed`; #1281), Ollama's `GET /api/version`, `GET /api/tags` and `POST /api/show` (`ollama.go`, for a client that takes a custom endpoint only as an Ollama server: Visual Studio's and VS Code's Copilot, wani on Discord; the list is `catalogFor`'s, so a key's models alone, `/api/show` tells `capabilities` (`completion`, `tools`, `vision` from `Images`, `thinking` from `Thinks`) and `model_info` with `general.architecture` `magpie`, `general.basename`, `magpie.context_length` (32768, VS Code's own default, when the window isn't known, since VS Code would take a 0) and the `general.parameter_count` and `magpie.embedding_length` the ollama CLI before 0.7 reads unchecked; a name ending `:latest`, as the ollama CLI asks, is the bare id unless a model is really called that (`ollamaName`); the client then chats on `/v1/chat/completions`, or on Ollama's own `POST /api/chat` and `POST /api/generate` (`ollama_chat.go`, Zed's Ollama provider, super_lalalalala on Discord), which `ollamaChatBody` turns into a `/v1/chat/completions` body handed to the same `handle(provider.Chat)` on a clone of the request, so they are routed, held to a key's models and counted as it is: `messages` with `images` (base64, its type sniffed, as data-URL parts; a tool message's in a user message after it), an assistant's `thinking` as `reasoning_content` and its `tool_calls` given ids where Ollama sent none, a tool result matched to its call by `tool_call_id`, else `tool_name`, else in order (one with no call is a 400), `tools`, `format` (`json` or a schema), `think` (a level is `reasoning_effort`), and `options`' `temperature`, `top_p`, `seed`, `stop`, the two penalties and a positive `num_predict` as `max_tokens` (`num_ctx`, `top_k` and the rest of Ollama's own are dropped); `ollamaWriter` turns the reply into Ollama's NDJSON lines (streamed unless `stream:false`, then one object), tool calls with their `arguments` as an object and an `id`, and a last `done` line with `done_reason` and the token counts, and an error into `{"error":"…"}` with the status `/v1` would give; a request with no messages or prompt is answered `done_reason:"load"`, as Ollama does to warm a model), `/mcp/{name}`, `/mcp/magpie/web-search` and `/mcp/magpie/vision` (magpie's own web search and image recognition as MCP servers, below) and the `/v1/magpie/*` status endpoints. `/v1/magpie/heap` (`heapProfile`, [`heap.go`](../../internal/gateway/heap.go), for `magpie heap`) is held tighter than the rest by `heapRefused` in the handler itself, so it holds where `Handler` is served without `lanGuard` too: only a loopback caller that isn't proxied (`local`), carries no browser mark (`fromBrowser`) and names a loopback `Host` (`loopbackHost`) gets the profile; a gateway key never lets anything else in. `/v1/magpie/cpu` (`cpuProfile`, [`cpu.go`](../../internal/gateway/cpu.go), for `magpie cpu`) records a CPU profile for `?seconds=` (30, from 5 to 120) and is held to the same callers by `cpuRefused`, which shares `profileRefused` with `heapRefused`; one profile records at a time (`cpuRecording`), a second is answered 409, and a caller that hangs up stops it. `/v1/magpie/goroutines` (`goroutineDump`, [`goroutines.go`](../../internal/gateway/goroutines.go), for `magpie goroutines`, #3362) answers every goroutine's stack and wait as text (`runtime/pprof`'s `goroutine` profile at debug 2) to the same callers, through `goroutinesRefused`. `/v1/alpha/search` (also without `/v1`, and on the Codex backend path for a magpie model) is Codex's web.run search (`alphaSearch`, `codex_search.go`): a ChatGPT account's model goes to the ChatGPT backend's `/alpha/search` on the account's sign-in, the next account asked on 401, 403, 429 or 5xx; any other model's `search_query` is searched by `webSearch`, and when nothing answers the reason is a 200 with `output`, since Codex ends the turn on any failed web.run (anton on Discord). A GET on `/v1/responses` (a client opening Responses over a WebSocket) is answered 426, a plain one 405: Responses is relayed over HTTP only. Every route is wrapped in `counted`, `corsGuard`, `callerGuard`, `withCaller` and `keyLimited`; `/v1/messages`, `/messages` and `/v1/messages/count_tokens` in `keepsSignIn` too. `corsGuard` (#1051) serves CORS only to the origins listed in Settings › Network › Web pages (`settings.CORSOrigins`, cleaned to `scheme://host[:port]` by `settings.CleanOrigins`, which refuses a wildcard). A browser extension's own pages, popup and service worker are listed the same way by their origin, which is the browser's scheme and the extension's id: `chrome-extension://<id>` (Chromium, the id 32 letters a–p), `moz-extension://<uuid>` (Firefox, the Internal UUID about:debugging shows, not the add-on id) and `safari-web-extension://<uuid>` (Safari, which picks a new UUID on each launch, so a listing lasts until Safari restarts); the id must have that browser's shape, cased any way, and nothing else is taken — no bare scheme, no wildcard, no path (SkyAerope on Discord: an extension without host permissions is held to CORS like a page). A content script's fetch carries the page's origin, not the extension's. A listed extension is treated exactly as a listed page, below; one not listed passes as before (other schemes, below) with no CORS headers. For a listed origin it answers the preflight (204) and adds `Access-Control-Allow-Origin`, and every other request from it needs an enabled gateway key (`access.Authenticate`), even on loopback, or gets a 401. It takes the key from the caller `lanGuard` recorded too, since on the running server `lanGuard` has already put magpie's own token in the key's place. Any other web page is refused 403 on every route (`foreignPage`): on loopback a call needs no key and the body's Content-Type isn't checked, so a page the user visits could otherwise send a "simple" `text/plain` POST and spend their subscriptions blind. Only an `http`/`https` Origin is judged, and one on this computer (`localhost`, `*.localhost`, `127.0.0.0/8`, `[::1]`, any port) or the request's own origin (its host and port equal to `Host`, such as a page on the gateway's LAN address) passes, as does usemagpie.ai on `/v1/magpie/omarchy`. Other schemes (`app://`, `file://`, `tauri://`, `vscode-webview://`, `chrome-extension://`) come from desktop apps, webviews and extensions that don't enforce CORS, and pass keyless as before; `null` is not one of them, since any web page sends it from a sandboxed frame and an opened file sends it too, so a call with it needs an enabled gateway key as a listed origin's does, and is answered 403 without one (no CORS headers either way, so a frame or a local page still reads no reply). A page at a subdomain of localhost (`localhostSubdomain`, #2256) needs a key the same way: Chrome and Firefox resolve `*.localhost` to loopback themselves, but Safari and other WebKit views ask the system resolver, which before macOS 26 asks the network's DNS (WebKit bug 160504), so a resolver that answers serves such a page from anywhere. `tauri.localhost` and `wails.localhost` stay keyless on Windows only, where Tauri's and Wails' webviews serve themselves at those names; on macOS and Linux those apps use `tauri://` and `wails://`. A request with no Origin (every agent's) passes through with no CORS headers, as before. A page whose own name resolves to 127.0.0.1 (DNS rebinding) would count as same-origin here, so `lanGuard` refuses it first (`rebound`, 403): a loopback request carrying a browser's marks (`Origin`, `Sec-Fetch-Site`/`Sec-Fetch-Mode`, a `Mozilla/` user agent) whose `Host` isn't this computer's (an IP, `localhost` but not a subdomain of it, which WebKit asks the DNS for (#2256), a container runtime's name for the host such as `host.docker.internal`, `MAGPIE_PUBLIC_URL`'s or `MAGPIE_ADDR`'s host, the machine's own name) needs an enabled gateway key. Agents send none of those marks and pass under any name. Before any of that, `lanGuard` lets in one request without a key from anywhere: `POST /_magpie/claude-mcp/<token>` with the token of a Claude subscription run now live (`ownCallback`, see [the Claude subscription bridge](claude-subscription-bridge.md)), the run's MCP helper calling back to a `MAGPIE_ADDR` that isn't loopback (#1546). Next, before the key or `rebound` is looked at, the allow list (lc on Discord, magpie in Docker on a public host): with `MAGPIE_ALLOW_IPS` or Settings › Network › Allowed addresses (`settings.AllowIPs`, this computer's own, never brought by a sync or restore: see [sync and backup](sync-backup.md#what-stays-on-each-computer); the variable wins and the page then only shows it) naming IPs and CIDR ranges, a request whose connection (`RemoteAddr`, IPv4-mapped IPv6 unmapped, zone dropped) is outside them is refused 403 naming its address, even with an enabled gateway key (`refuseNotAllowed`). Loopback and this computer's own interface addresses always pass. A forwarding header is never taken for the address, so a proxy or tunnel on this computer passes (it arrives from loopback) and its clients are limited at the proxy. An entry that doesn't parse lets nothing in and is logged once; the list stays on. Settings' list is read with `settings.LoadChecked`: a settings.json that can't be read, or doesn't parse with no last good generation, is unknown, never "no list", so the list read last in this process stays in force (`lastAllow`), and with none read yet only this computer passes, the 403 saying settings.json couldn't be read; logged once. No file, or an empty one, is no list. The Settings page's saves read with `LoadChecked` too and refuse such a file, as `settings.Save` does. `magpie web`'s server is wrapped in the same check (`AllowGuard`), and its port hands the gateway's client routes (`ClientPath`) to `Server.Front`, the same lanGuard-over-Handler the gateway's listener serves, so every route there is judged as on the gateway's port (`TestClientPathCoversHandler` fails when a route Handler adds is in neither `ClientPath` nor its list of routes that stay on the gateway's port). Settings refuses a list that would leave out the address the page saving it is open from (`KeepsOut`); the page names that address, which behind Docker or a proxy is the proxy's, and offers to add it to the refused list (`allowSays`, Mathresearcher on Discord). | [`gateway.go`](../../internal/gateway/gateway.go), [`allow.go`](../../internal/gateway/allow.go), [`internal/settings/allowips.go`](../../internal/settings/allowips.go), [`cors.go`](../../internal/gateway/cors.go), [`codex_catalog.go`](../../internal/gateway/codex_catalog.go), [`ollama.go`](../../internal/gateway/ollama.go), [`ollama_chat.go`](../../internal/gateway/ollama_chat.go), [`internal/settings/origins.go`](../../internal/settings/origins.go) |
+| HTTP surface | `Handler` routes `/v1/chat/completions`, `/v1/responses`, `/v1/messages` (each also without `/v1`), `/v1/models`, Gemini's `/v1beta/models/…`, the Codex backend path, `/v1/codex/models` (Codex's catalog shape for a Codex elsewhere with a gateway key, `codexCatalog`, kept to the key's models by `keyAllowed`; #1281), Ollama's `GET /api/version`, `GET /api/tags` and `POST /api/show` (`ollama.go`, for a client that takes a custom endpoint only as an Ollama server: Visual Studio's and VS Code's Copilot, wani on Discord; the list is `catalogFor`'s, so a key's models alone, `/api/show` tells `capabilities` (`completion`, `tools`, `vision` from `Images`, `thinking` from `Thinks`) and `model_info` with `general.architecture` `magpie`, `general.basename`, `magpie.context_length` (32768, VS Code's own default, when the window isn't known, since VS Code would take a 0) and the `general.parameter_count` and `magpie.embedding_length` the ollama CLI before 0.7 reads unchecked; a name ending `:latest`, as the ollama CLI asks, is the bare id unless a model is really called that (`ollamaName`); the client then chats on `/v1/chat/completions`, or on Ollama's own `POST /api/chat` and `POST /api/generate` (`ollama_chat.go`, Zed's Ollama provider, super_lalalalala on Discord), which `ollamaChatBody` turns into a `/v1/chat/completions` body handed to the same `handle(provider.Chat)` on a clone of the request, so they are routed, held to a key's models and counted as it is: `messages` with `images` (base64, its type sniffed, as data-URL parts; a tool message's in a user message after it), an assistant's `thinking` as `reasoning_content` and its `tool_calls` given ids where Ollama sent none, a tool result matched to its call by `tool_call_id`, else `tool_name`, else in order (one with no call is a 400), `tools`, `format` (`json` or a schema), `think` (a level is `reasoning_effort`), and `options`' `temperature`, `top_p`, `seed`, `stop`, the two penalties and a positive `num_predict` as `max_tokens` (`num_ctx`, `top_k` and the rest of Ollama's own are dropped); `ollamaWriter` turns the reply into Ollama's NDJSON lines (streamed unless `stream:false`, then one object), tool calls with their `arguments` as an object and an `id`, and a last `done` line with `done_reason` and the token counts, and an error into `{"error":"…"}` with the status `/v1` would give; a request with no messages or prompt is answered `done_reason:"load"`, as Ollama does to warm a model), `/mcp/{name}`, `/mcp/magpie/web-search`, `/mcp/magpie/vision` and `/mcp/magpie/image` with its `/mcp/magpie/image/files/{name}` (magpie's own web search, image recognition and image generation as MCP servers, below) and the `/v1/magpie/*` status endpoints. `/v1/magpie/heap` (`heapProfile`, [`heap.go`](../../internal/gateway/heap.go), for `magpie heap`) is held tighter than the rest by `heapRefused` in the handler itself, so it holds where `Handler` is served without `lanGuard` too: only a loopback caller that isn't proxied (`local`), carries no browser mark (`fromBrowser`) and names a loopback `Host` (`loopbackHost`) gets the profile; a gateway key never lets anything else in. `/v1/magpie/cpu` (`cpuProfile`, [`cpu.go`](../../internal/gateway/cpu.go), for `magpie cpu`) records a CPU profile for `?seconds=` (30, from 5 to 120) and is held to the same callers by `cpuRefused`, which shares `profileRefused` with `heapRefused`; one profile records at a time (`cpuRecording`), a second is answered 409, and a caller that hangs up stops it. `/v1/magpie/goroutines` (`goroutineDump`, [`goroutines.go`](../../internal/gateway/goroutines.go), for `magpie goroutines`, #3362) answers every goroutine's stack and wait as text (`runtime/pprof`'s `goroutine` profile at debug 2) to the same callers, through `goroutinesRefused`. `/v1/alpha/search` (also without `/v1`, and on the Codex backend path for a magpie model) is Codex's web.run search (`alphaSearch`, `codex_search.go`): a ChatGPT account's model goes to the ChatGPT backend's `/alpha/search` on the account's sign-in, the next account asked on 401, 403, 429 or 5xx; any other model's `search_query` is searched by `webSearch`, and when nothing answers the reason is a 200 with `output`, since Codex ends the turn on any failed web.run (anton on Discord). A GET on `/v1/responses` (a client opening Responses over a WebSocket) is answered 426, a plain one 405: Responses is relayed over HTTP only. Every route is wrapped in `counted`, `corsGuard`, `callerGuard`, `withCaller` and `keyLimited`; `/v1/messages`, `/messages` and `/v1/messages/count_tokens` in `keepsSignIn` too. `corsGuard` (#1051) serves CORS only to the origins listed in Settings › Network › Web pages (`settings.CORSOrigins`, cleaned to `scheme://host[:port]` by `settings.CleanOrigins`, which refuses a wildcard). A browser extension's own pages, popup and service worker are listed the same way by their origin, which is the browser's scheme and the extension's id: `chrome-extension://<id>` (Chromium, the id 32 letters a–p), `moz-extension://<uuid>` (Firefox, the Internal UUID about:debugging shows, not the add-on id) and `safari-web-extension://<uuid>` (Safari, which picks a new UUID on each launch, so a listing lasts until Safari restarts); the id must have that browser's shape, cased any way, and nothing else is taken — no bare scheme, no wildcard, no path (SkyAerope on Discord: an extension without host permissions is held to CORS like a page). A content script's fetch carries the page's origin, not the extension's. A listed extension is treated exactly as a listed page, below; one not listed passes as before (other schemes, below) with no CORS headers. For a listed origin it answers the preflight (204) and adds `Access-Control-Allow-Origin`, and every other request from it needs an enabled gateway key (`access.Authenticate`), even on loopback, or gets a 401. It takes the key from the caller `lanGuard` recorded too, since on the running server `lanGuard` has already put magpie's own token in the key's place. Any other web page is refused 403 on every route (`foreignPage`): on loopback a call needs no key and the body's Content-Type isn't checked, so a page the user visits could otherwise send a "simple" `text/plain` POST and spend their subscriptions blind. Only an `http`/`https` Origin is judged, and one on this computer (`localhost`, `*.localhost`, `127.0.0.0/8`, `[::1]`, any port) or the request's own origin (its host and port equal to `Host`, such as a page on the gateway's LAN address) passes, as does usemagpie.ai on `/v1/magpie/omarchy`. Other schemes (`app://`, `file://`, `tauri://`, `vscode-webview://`, `chrome-extension://`) come from desktop apps, webviews and extensions that don't enforce CORS, and pass keyless as before; `null` is not one of them, since any web page sends it from a sandboxed frame and an opened file sends it too, so a call with it needs an enabled gateway key as a listed origin's does, and is answered 403 without one (no CORS headers either way, so a frame or a local page still reads no reply). A page at a subdomain of localhost (`localhostSubdomain`, #2256) needs a key the same way: Chrome and Firefox resolve `*.localhost` to loopback themselves, but Safari and other WebKit views ask the system resolver, which before macOS 26 asks the network's DNS (WebKit bug 160504), so a resolver that answers serves such a page from anywhere. `tauri.localhost` and `wails.localhost` stay keyless on Windows only, where Tauri's and Wails' webviews serve themselves at those names; on macOS and Linux those apps use `tauri://` and `wails://`. A request with no Origin (every agent's) passes through with no CORS headers, as before. A page whose own name resolves to 127.0.0.1 (DNS rebinding) would count as same-origin here, so `lanGuard` refuses it first (`rebound`, 403): a loopback request carrying a browser's marks (`Origin`, `Sec-Fetch-Site`/`Sec-Fetch-Mode`, a `Mozilla/` user agent) whose `Host` isn't this computer's (an IP, `localhost` but not a subdomain of it, which WebKit asks the DNS for (#2256), a container runtime's name for the host such as `host.docker.internal`, `MAGPIE_PUBLIC_URL`'s or `MAGPIE_ADDR`'s host, the machine's own name) needs an enabled gateway key. Agents send none of those marks and pass under any name. Before any of that, `lanGuard` lets in one request without a key from anywhere: `POST /_magpie/claude-mcp/<token>` with the token of a Claude subscription run now live (`ownCallback`, see [the Claude subscription bridge](claude-subscription-bridge.md)), the run's MCP helper calling back to a `MAGPIE_ADDR` that isn't loopback (#1546). Next, before the key or `rebound` is looked at, the allow list (lc on Discord, magpie in Docker on a public host): with `MAGPIE_ALLOW_IPS` or Settings › Network › Allowed addresses (`settings.AllowIPs`, this computer's own, never brought by a sync or restore: see [sync and backup](sync-backup.md#what-stays-on-each-computer); the variable wins and the page then only shows it) naming IPs and CIDR ranges, a request whose connection (`RemoteAddr`, IPv4-mapped IPv6 unmapped, zone dropped) is outside them is refused 403 naming its address, even with an enabled gateway key (`refuseNotAllowed`). Loopback and this computer's own interface addresses always pass. A forwarding header is never taken for the address, so a proxy or tunnel on this computer passes (it arrives from loopback) and its clients are limited at the proxy. An entry that doesn't parse lets nothing in and is logged once; the list stays on. Settings' list is read with `settings.LoadChecked`: a settings.json that can't be read, or doesn't parse with no last good generation, is unknown, never "no list", so the list read last in this process stays in force (`lastAllow`), and with none read yet only this computer passes, the 403 saying settings.json couldn't be read; logged once. No file, or an empty one, is no list. The Settings page's saves read with `LoadChecked` too and refuse such a file, as `settings.Save` does. `magpie web`'s server is wrapped in the same check (`AllowGuard`), and its port hands the gateway's client routes (`ClientPath`) to `Server.Front`, the same lanGuard-over-Handler the gateway's listener serves, so every route there is judged as on the gateway's port (`TestClientPathCoversHandler` fails when a route Handler adds is in neither `ClientPath` nor its list of routes that stay on the gateway's port). Settings refuses a list that would leave out the address the page saving it is open from (`KeepsOut`); the page names that address, which behind Docker or a proxy is the proxy's, and offers to add it to the refused list (`allowSays`, Mathresearcher on Discord). | [`gateway.go`](../../internal/gateway/gateway.go), [`allow.go`](../../internal/gateway/allow.go), [`internal/settings/allowips.go`](../../internal/settings/allowips.go), [`cors.go`](../../internal/gateway/cors.go), [`codex_catalog.go`](../../internal/gateway/codex_catalog.go), [`ollama.go`](../../internal/gateway/ollama.go), [`ollama_chat.go`](../../internal/gateway/ollama_chat.go), [`internal/settings/origins.go`](../../internal/settings/origins.go) |
| Model resolution | `serve` resolves the model the agent asked for: Claude tier stand-ins, a routing group (`provider.GroupFor`: its id, the model's id however a vendor spells it, else the one group whose name it is, unless a provider serves a model by that id; the user's own groups only while the `groups` plugin is on, [the plugin runtime](plugin-runtime.md#gateway-extension-points-p3-router-strategy)), auto stand-ins, the agent's own stand-in (`StandIn`: Codex's config.toml `model` when magpie is its provider, for a name it sends that magpie doesn't serve, such as its auto-review's `codex-auto-review`), itself taken through `GroupFor` and auto stand-ins so a config naming a group bare reaches it, then `provider.Resolve`. A Codex memory call or auto-review (`x-openai-subagent: guardian`, or `codex-auto-review`) that still names no served model goes to the model Codex's last turn was answered on (`codexMemoryStandIn`, per agent and gateway key), so a Codex on magpie through another provider table, a profile or another computer isn't answered 404 (Adam on Discord, `TestCodexAutoReviewReachesItsTurnsModel`). A bare model id several providers serve (MOMO on Discord: `glm-5.3` under opencode-go and a6api) is never refused: with found groups on (the default) `GroupFor` takes it to the group magpie found for it (`auto-<slug>`), its members in the Providers order, routed and failed over as any group, a conversation staying with the member that answered it while its cache is worth it (affinity); with them off, `Resolve` takes the first provider in the Providers order that exposes it, with no fallback. magpie writes only `provider/model` ids into agents' configs (`magpieModels`), so a bare id is one the user wrote. `/v1/models` lists bare ids only when the user asks for them ([Model ids without the provider](#model-ids-without-the-provider)). Every reply says who answered in `X-Magpie-Provider` and `X-Magpie-Model` (`noteMember`), and the usage log and the Routing page keep it. A model id with `~` and no slash that resolves as nothing else is the one served id (a provider's model or a group) whose slashes it spells as `~` (`unflat`, `provider.Unflat`; MiniMax Code's spelling, #1387), taken back in `validateModel` before key limits, middleware and usage see it; two ids spelled alike are neither. A Claude model id whose version is dashed (`relay/claude-opus-4-6-us`, with or without `:level` and `[1m]`) that is not served as spelled is the one served id it dashes (`provider.ClaudeUndashed`; the spelling magpie gives Claude Code, which reads a dotted version as Claude Opus 4), taken back in `validateModel` and `unprefixed` the same way. An unknown model gets a 404 naming why: a switched-off provider, a disabled group or an empty group. Otherwise it names up to three ids nearest to what was asked (`provider.Closest`: words of a model's display name count before words of its id, and a provider's id or name only alongside them, so "Zen-Space-Bunny" is told `opencode-zen-free/space-bunny-free`, MOMO on Discord), then the ways a model can be named, then every id (`unknownModel`; chat and embeddings). | [`gateway.go`](../../internal/gateway/gateway.go), [`internal/provider/group.go`](../../internal/provider/group.go), [`internal/provider/flat.go`](../../internal/provider/flat.go), [`internal/provider/claudedash.go`](../../internal/provider/claudedash.go) |
| A found group's two ids | A group magpie found is listed (`/v1/models`, the catalog, the lists written into agents, the pickers, the CLI and TUI) as `auto/<model>`: the model's id as a vendor spells it, dots kept and with no vendor prefix (`auto/claude-opus-5.5`; `provider.Group.Ref`, `spelledIn`, preferring a spelling models.dev lists), so a client that looks the part after the slash up on models.dev finds it (ZeroClover on X). Its old id `group/auto-<slug>` is served forever. Inside magpie the group keeps that one id: `validateModel` takes `auto/<model>` (with `:level` or `[1m]`) back to it with `provider.CanonicalRef` before key limits, routing, affinity and the usage log see the request, and the request body is rewritten to it, so both ids count as one group. A gateway key naming either id, `group/*` or `auto/*` passes for both (`allows` in `key_models.go`). `SaveGroup` stores a member, rule or classifier picked as `auto/<model>` by the old id. An agent field magpie set to `group/auto-<slug>` moves to `auto/<model>` on the next `SyncCatalog` (`agent.renameFoundGroups`), but only while its value is still what applied.json says magpie wrote; a value the user set or changed stays. A pick of the group by its old id written now (`Agent.Apply`) is written as `auto/<model>`. An agent's hidden, picked and ordered models saved by the old id hold for the new one (`PublicRef` in `visible.go`), and a bare model id still reaches its found group (`bareTarget`). Claude Code is given a dotted Claude version dashed, so it may send `auto/claude-sonnet-4-5`: that is the same group, and `Reseat` still moves an agent off it once the group is gone (`samePick`). `auto` is no provider id: `Save` and `Rename` refuse it for a new provider, and a plugin's `auto` is `auto-plugin` (`PluginID`). A provider of the user's saved as `auto` before keeps its models: while it exists, `auto/<x>` is its model and found groups list as `group/auto-<slug>` (`autoRefsOn`). Hand-made groups stay `group/<id>`. | [`internal/provider/autoref.go`](../../internal/provider/autoref.go), [`key_models.go`](../../internal/gateway/key_models.go), [`internal/agent/autoref.go`](../../internal/agent/autoref.go) |
| Candidates | `plan` lists one provider's candidates: one per key or account on, then each `Fallback` model, with resting ones last (`restLast`). `planGroup` and `planLevel` weigh a group's members together. A nested group plans by its own routing (#576). So does a member whose provider routes other than Smart (`ownRouting`): weighed as one by its first not resting (`firstAwake`), tried whole in its own order; `leastHeld` leaves such a group's order alone too. A provider on Smart, or routed as the group is, is pooled with the group's other keys and accounts. `group_own_routing_test.go`. A member's own fallbacks are not the group's. | [`fallback.go`](../../internal/gateway/fallback.go) |
@@ -723,6 +723,44 @@ Covered by `TestVisionMCPAnalyzesImage`, `TestVisionMCPWithNoSeer`,
`TestVisionMCPGuarded`, `TestVisionMCPHeldToKeyLimit` and
`vision-mcp.test.cjs`.
+## Image generation as an MCP server
+
+`ImageMCPPath`, `/mcp/magpie/image` ([`image_mcp.go`](../../internal/gateway/image_mcp.go)),
+serves Settings › Images › Image generation as a Streamable HTTP MCP server
+with the tools the stdio Magpie Image server has (`internal/imagemcp`):
+`generate_image`, `generate_video` and `generation_result` (JonahFu on X:
+magpie in Docker in gateway mode has no Library, so no stdio server). Their
+schemas are the gateway's `ImageToolProps`, `VideoToolProps` and
+`ResultToolProps`, which the stdio server takes too, adding only its `path`
+argument. It is served by `serveMCP`, so it is guarded as the vision server
+is: `local` or a shared magpie's caller, a refused Origin 403, POST only, a
+32 MB body, and each call reserves the calling key's limit. An input image
+goes through `mcpImageSrc`, the vision server's reader: a URL or data URL as
+it is, a file path only for a `local` caller.
+
+A drawing runs `images` in-process on a clone of the request
+(`inProcess`, the caller and key kept), so it is routed, counted and held to
+the key's models as `/v1/images/generations` and `/v1/images/edits` are. It
+runs as a job under `context.WithoutCancel` for at most 15 minutes; the tool
+answers within 20 seconds, else with an id that `generation_result` waits on.
+Each image comes back as MCP image content (up to 5 MB each) and at a URL
+under `/mcp/magpie/image/files/` (a vendor's URL is fetched first). The
+file store is in memory, kept an hour and 256 MB at most, oldest first out.
+A file and a job belong to the gateway key that made them: another key gets
+404 (`KeyID`). The URL's base is `MAGPIE_PUBLIC_URL` when set, else the
+request's scheme and `Host`. A video is started through `videosCreate` and
+polled through `videosGet` in-process; when done the tool gives
+`/v1/videos/<id>/content`, which needs the same key. Image generation Off
+(`settings.ImageGen == "off"`) is a tool error that says where to pick a
+model, checked before the arguments. Nothing is written into the agent's
+project. Settings › Models shows the address (`#imageMcpRow`, below
+`#visionMcpRow`).
+
+Covered by `TestImageMCPDrawsAndServesTheImage`, `TestImageMCPSaysWhenOff`,
+`TestImageMCPGuarded`, `TestImageMCPHeldToKeyLimit`,
+`TestImageMCPSlowDrawingIsWaitedOn`, `TestImageMCPMakesVideos`,
+`TestImageMCPIsServed` and `image-mcp.test.cjs`.
+
## Side models magpie picks: allowance first
When magpie picks the model itself (`AutoVision` for image recognition,
diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go
index 4a12e7dba..cf34c99d8 100644
--- a/internal/gateway/gateway.go
+++ b/internal/gateway/gateway.go
@@ -389,6 +389,8 @@ type Server struct {
// from, so every later turn goes straight to one that takes them.
unfit map[string]bool
subscription *subscriptionBridge
+ // imageStore is the image MCP server's jobs and images (image_mcp.go)
+ imageStore imageMCPStore
debug bool
trace trace // what routing did with each request, for the Gateway view
titlePrompts titlePrompts
@@ -744,6 +746,8 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("/mcp/{name}", s.mcpProxy)
mux.HandleFunc(SearchMCPPath, s.searchMCP)
mux.HandleFunc(VisionMCPPath, s.visionMCP)
+ mux.HandleFunc(ImageMCPPath, s.imageMCP)
+ mux.HandleFunc(ImageMCPFiles+"{name}", s.imageMCPFile)
mux.HandleFunc(ToolsMCPPath, s.toolsMCP)
mux.HandleFunc(CodexPath+"/", s.codexBackend)
mux.HandleFunc("GET "+CodexCatalogPath, s.codexCatalog)
@@ -779,7 +783,7 @@ var catalogSize stamped[struct{}, int]
func (s *Server) info(w http.ResponseWriter, r *http.Request) {
out := map[string]any{"name": "magpie", "version": Version, "models": catalogSize.get(struct{}{}, func() int { return len(provider.Catalog()) }), "window": Window,
- "apis": []string{"/v1/chat/completions", "/v1/responses", "/v1/messages", "/v1/systemone", "/v1beta/models/{model}:generateContent", "/v1/images/generations", "/v1/images/edits", "/v1/videos", "/v1/embeddings", "/v1/rerank", "/v1/audio/transcriptions", "/v1/audio/translations", "/v1/magpie/quotas", "/v1/magpie/quotas/history", "/v1/magpie/usage", "/v1/magpie/usage/requests", "/v1/magpie/route", "/v1/magpie/trace", SearchMCPPath, VisionMCPPath, ToolsMCPPath}}
+ "apis": []string{"/v1/chat/completions", "/v1/responses", "/v1/messages", "/v1/systemone", "/v1beta/models/{model}:generateContent", "/v1/images/generations", "/v1/images/edits", "/v1/videos", "/v1/embeddings", "/v1/rerank", "/v1/audio/transcriptions", "/v1/audio/translations", "/v1/magpie/quotas", "/v1/magpie/quotas/history", "/v1/magpie/usage", "/v1/magpie/usage/requests", "/v1/magpie/route", "/v1/magpie/trace", SearchMCPPath, VisionMCPPath, ImageMCPPath, ToolsMCPPath}}
// which process serves, for another magpie on this computer to name it
// on its Routing page (#2606); not told to another machine
if local(r) {
diff --git a/internal/gateway/image_mcp.go b/internal/gateway/image_mcp.go
new file mode 100644
index 000000000..bc613f7ad
--- /dev/null
+++ b/internal/gateway/image_mcp.go
@@ -0,0 +1,710 @@
+package gateway
+
+import (
+ "bytes"
+ "cmp"
+ "context"
+ "crypto/rand"
+ "encoding/base64"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "net/http"
+ "strconv"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/yetone/magpie/internal/access"
+ "github.com/yetone/magpie/internal/settings"
+)
+
+// magpie's image generation, the model Settings › Images › Image generation
+// names, is also an MCP server the gateway serves itself, beside image
+// recognition's (JonahFu on X): Magpie Image (internal/imagemcp) is a stdio
+// server the Library gives an agent on this computer, which a gateway in
+// Docker, or an agent on another computer, has none of. Streamable HTTP at
+// ImageMCPPath, served and guarded as VisionMCPPath is (serveMCP), with
+// Magpie Image's tools and arguments (ImageToolProps, VideoToolProps)
+// but where to save: the gateway can't write into the agent's project, so
+// an image comes back in the answer, as MCP image content, and at a URL
+// the gateway keeps it at for an hour (ImageMCPFiles); a video at the
+// gateway's own /v1/videos/{id}/content. What draws is the images API
+// itself (s.images, s.videosCreate), called in-process with the MCP
+// request's caller, so a gateway key's models, limits and usage hold it
+// as they hold /v1/images.
+
+// ImageMCPPath is where the image generation MCP server is served.
+const ImageMCPPath = "/mcp/magpie/image"
+
+// ImageMCPFiles is where the images it made are served from.
+const ImageMCPFiles = ImageMCPPath + "/files/"
+
+// The tools Magpie Image and the image MCP server have.
+const (
+ GenerateImageTool = "generate_image"
+ GenerateVideoTool = "generate_video"
+ GenerationResultTool = "generation_result"
+)
+
+// ImageToolProps is generate_image's arguments but where to save: images
+// says how an image to draw from may be given.
+func ImageToolProps(images string) map[string]any {
+ return map[string]any{
+ "prompt": map[string]any{"type": "string", "description": "What to draw, or how to change the reference images."},
+ "size": map[string]any{"type": "string", "description": "WIDTHxHEIGHT (1024x1024, 1536x1024, 1024x1536) or an aspect ratio (16:9). Default: the model's."},
+ "n": map[string]any{"type": "integer", "minimum": 1, "maximum": 4, "description": "How many images (default 1)."},
+ "reference_images": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "Images to edit or draw from: " + images + "."},
+ "quality": map[string]any{"type": "string", "description": "low, medium, high or auto, for models that take it."},
+ "background": map[string]any{"type": "string", "description": "transparent, opaque or auto, for models that take it."},
+ }
+}
+
+// VideoToolProps is generate_video's arguments but where to save: image
+// says how the image to animate may be given, images how those its
+// subjects are drawn from may.
+func VideoToolProps(image, images string) map[string]any {
+ return map[string]any{
+ "prompt": map[string]any{"type": "string", "description": "What happens in the video: the subject, its motion, the camera."},
+ "seconds": map[string]any{"type": "integer", "minimum": 1, "maximum": 15, "description": "Length in whole seconds (1-15). Default: the model's (about 8)."},
+ "size": map[string]any{"type": "string", "description": "An aspect ratio (16:9, 9:16, 1:1, 4:3, 3:4, 3:2, 2:3) or WIDTHxHEIGHT (1280x720; its shorter side picks 480p, 720p or 1080p). Default: the model's."},
+ "image": map[string]any{"type": "string", "description": "An image to animate, the video's first frame: " + image + "."},
+ "reference_images": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "Images the video's subjects are drawn from: " + images + "."},
+ }
+}
+
+// ResultToolProps is generation_result's arguments.
+func ResultToolProps() map[string]any {
+ return map[string]any{
+ "id": map[string]any{"type": "string", "description": "The id " + GenerateImageTool + " or " + GenerateVideoTool + " answered with."},
+ }
+}
+
+// imageMCPWithin is how long a call waits for its image or video before it
+// answers with an id to wait on (generation_result), as Magpie Image's
+// answerWithin: agents give a tool call as little as 30 s (#575).
+var imageMCPWithin = 20 * time.Second
+
+// imageMCPPoll is how often a call asks after its video.
+var imageMCPPoll = 3 * time.Second
+
+const (
+ // imageMCPKeep is how long an image's file, and a job, is kept.
+ imageMCPKeep = time.Hour
+ // imageMCPFilesMost is how many bytes of images are kept at once; the
+ // oldest go first.
+ imageMCPFilesMost = 256 << 20
+ // imageMCPInline is the largest image put in the answer itself.
+ imageMCPInline = 5 << 20
+ // imageMCPJobMost is how long a job may draw.
+ imageMCPJobMost = 15 * time.Minute
+)
+
+const imageMCPFrom = "an http(s) URL or a data URL (data:image/png;base64,...); an absolute file path only for an agent on the computer magpie runs on"
+
+func imageMCPTools() []any {
+ ro := func(props map[string]any) map[string]any {
+ return map[string]any{"type": "object", "properties": props, "required": []string{"prompt"}}
+ }
+ return []any{
+ map[string]any{
+ "name": GenerateImageTool,
+ "title": "Generate image",
+ "description": "Generate an image from a text prompt, or edit/combine images given as reference_images, with the image model set in Magpie (Settings → Images → Image generation). " +
+ "The image comes back in the answer and at a URL Magpie keeps it at for an hour (from another computer, download it with the same Authorization header this server is called with); nothing is saved in your project, so save it there yourself if you need it. " +
+ "When it is not ready within about 20 seconds, the answer is an id to wait on with " + GenerationResultTool + " instead. Write a detailed prompt: subject, style, composition, colours, any text to render exactly.",
+ "inputSchema": ro(ImageToolProps(imageMCPFrom + "s")),
+ "annotations": map[string]any{"readOnlyHint": false, "destructiveHint": false, "openWorldHint": true},
+ },
+ map[string]any{
+ "name": GenerateVideoTool,
+ "title": "Generate video",
+ "description": "Generate a short video from a text prompt, optionally starting from an image to animate, with a video model set up in Magpie. It takes from ten seconds to a few minutes; " +
+ "the answer is the URL to download the mp4 from (with the same Authorization header this server is called with), or, when it is not ready within about 20 seconds, an id to wait on with " + GenerationResultTool + ". " +
+ "Nothing is saved in your project. Describe the subject, the motion and the camera: what moves, how, and what the shot is.",
+ "inputSchema": ro(VideoToolProps(imageMCPFrom, imageMCPFrom+"s")),
+ "annotations": map[string]any{"readOnlyHint": false, "destructiveHint": false, "openWorldHint": true},
+ },
+ map[string]any{
+ "name": GenerationResultTool,
+ "title": "Generation result",
+ "description": "Wait for an image or video that " + GenerateImageTool + " or " + GenerateVideoTool + " said is still being made, by the id it gave, and get it. " +
+ "It answers within about " + strconv.Itoa(int(imageMCPWithin/time.Second)) + " seconds: if it is still not ready, call this again with the same id (don't generate it again).",
+ "inputSchema": map[string]any{"type": "object", "properties": ResultToolProps(), "required": []string{"id"}},
+ "annotations": map[string]any{"readOnlyHint": true, "openWorldHint": false},
+ },
+ }
+}
+
+// imageMCPStore is what the image server keeps between calls: the jobs
+// still drawing or drawn within the hour, and the images they made.
+type imageMCPStore struct {
+ mu sync.Mutex
+ jobs map[string]*imageJob
+ files map[string]*imageFile
+ order []string // files, oldest first
+ bytes int
+ running sync.WaitGroup
+}
+
+// imageJob is one generate_image call's drawing: it goes on after the call
+// that started it has answered.
+type imageJob struct {
+ id, key string
+ started time.Time
+ done chan struct{}
+ result map[string]any // the tool result, once done
+}
+
+type imageFile struct {
+ data []byte
+ mime, key string
+ made time.Time
+}
+
+func (s *Server) imageMCP(w http.ResponseWriter, r *http.Request) {
+ serveMCP(w, r, "magpie's image generation", visionMCPBody, s.imageMCPCall)
+}
+
+// imageMCPCall answers one request: its result, or its JSON-RPC error.
+func (s *Server) imageMCPCall(r *http.Request, m mcpMessage) (any, map[string]any) {
+ switch m.Method {
+ case "initialize":
+ return map[string]any{
+ "protocolVersion": mcpVersion(m),
+ "capabilities": map[string]any{"tools": map[string]any{}},
+ "serverInfo": map[string]any{"name": "magpie-image", "title": "Magpie Image", "version": Version},
+ "instructions": GenerateImageTool + " draws with the image model set in magpie (Settings → Images → Image generation) and " + GenerateVideoTool + " makes a short video; images come back in the answer and at a URL, videos at a URL, nothing is saved in your project.",
+ }, nil
+ case "ping":
+ return map[string]any{}, nil
+ case "tools/list":
+ return map[string]any{"tools": imageMCPTools()}, nil
+ case "tools/call":
+ var p struct {
+ Name string `json:"name"`
+ Arguments json.RawMessage `json:"arguments"`
+ }
+ if err := json.Unmarshal(m.Params, &p); err != nil {
+ return nil, map[string]any{"code": -32602, "message": "invalid params: " + err.Error()}
+ }
+ switch p.Name {
+ case GenerateImageTool:
+ return s.imageMCPDraw(r, p.Arguments), nil
+ case GenerateVideoTool:
+ return s.imageMCPFilm(r, p.Arguments), nil
+ case GenerationResultTool:
+ var a struct {
+ ID string `json:"id"`
+ }
+ json.Unmarshal(p.Arguments, &a)
+ return s.imageMCPResult(r, strings.TrimSpace(a.ID)), nil
+ }
+ return nil, map[string]any{"code": -32602, "message": "unknown tool " + p.Name + ": this server has " + GenerateImageTool + ", " + GenerateVideoTool + " and " + GenerationResultTool}
+ }
+ return nil, map[string]any{"code": -32601, "message": "method not found: " + m.Method}
+}
+
+// imageGenOff is the tools' answer while Settings turned image generation
+// off, which turns video off with it.
+func imageGenOff() (map[string]any, bool) {
+ if settings.Load().ImageGen != "off" {
+ return nil, false
+ }
+ return toolResult("Image generation is off in magpie: pick a model for it in Settings → Images → Image generation", true), true
+}
+
+// imageMCPDraw starts generate_image's drawing and answers with what it
+// made, or, while it is still drawing after imageMCPWithin, with its id.
+func (s *Server) imageMCPDraw(r *http.Request, raw json.RawMessage) map[string]any {
+ if off, ok := imageGenOff(); ok {
+ return off
+ }
+ var a struct {
+ Prompt string `json:"prompt"`
+ Size string `json:"size"`
+ N int `json:"n"`
+ References []string `json:"reference_images"`
+ Quality string `json:"quality"`
+ Background string `json:"background"`
+ }
+ if err := json.Unmarshal(raw, &a); err != nil {
+ return toolResult("invalid arguments: "+err.Error(), true)
+ }
+ if strings.TrimSpace(a.Prompt) == "" {
+ return toolResult(GenerateImageTool+" needs a prompt", true)
+ }
+ req := map[string]any{"prompt": a.Prompt, "n": max(1, min(a.N, maxDrawings)), "response_format": "b64_json"}
+ for k, v := range map[string]string{"size": a.Size, "quality": a.Quality, "background": a.Background} {
+ if v != "" {
+ req[k] = v
+ }
+ }
+ edit := len(a.References) > 0
+ size := len(a.Prompt)
+ if edit {
+ var images []map[string]string
+ for _, ref := range a.References {
+ src, err := mcpImageSrc(r, strings.TrimSpace(ref), GenerateImageTool)
+ if err != nil {
+ return toolResult(err.Error(), true)
+ }
+ size += len(src)
+ images = append(images, map[string]string{"image_url": src})
+ }
+ req["images"] = images
+ }
+ who := access.Caller(r.Context())
+ release, err := reserveFor(who, int64(size), "", limitClock())
+ if err != nil {
+ return toolResult(err.Error(), true)
+ }
+ body, _ := json.Marshal(req)
+ path := "/v1/images/generations"
+ if edit {
+ path = "/v1/images/edits"
+ }
+ base := imageMCPBase(r)
+ // the drawing goes on when the call has answered: a job of its own,
+ // with the call's caller (its key, its agent) but not its end
+ in := inProcess(r, context.WithoutCancel(r.Context()), http.MethodPost, path, body)
+ j := s.imageMCPStart(who.KeyID, func() map[string]any {
+ defer release()
+ ctx, cancel := context.WithTimeout(in.Context(), imageMCPJobMost)
+ defer cancel()
+ rec := &replyRecorder{header: http.Header{}}
+ s.images(edit)(rec, in.WithContext(ctx))
+ return s.imageMCPDrawn(rec, who.KeyID, base)
+ })
+ return s.imageMCPAwait(r, j)
+}
+
+// imageMCPDrawn is the tool result of the images API's answer: each image
+// in it and at its URL.
+func (s *Server) imageMCPDrawn(rec *replyRecorder, key, base string) map[string]any {
+ var out struct {
+ Model string `json:"model"`
+ Data []struct {
+ B64 string `json:"b64_json"`
+ URL string `json:"url"`
+ Mime string `json:"mime_type"`
+ Revised string `json:"revised_prompt"`
+ } `json:"data"`
+ Text string `json:"text"`
+ }
+ json.Unmarshal(rec.body.Bytes(), &out)
+ if rec.code() != http.StatusOK {
+ return toolResult("no image: "+imageMCPSaid(rec.body.Bytes()), true)
+ }
+ if len(out.Data) == 0 {
+ return toolResult("no image came back", true)
+ }
+ var content []any
+ var urls, lost, revised []string
+ for i, d := range out.Data {
+ var img []byte
+ mt := d.Mime
+ var err error
+ switch {
+ case d.B64 != "":
+ if img, err = base64.StdEncoding.DecodeString(d.B64); err != nil {
+ lost = append(lost, fmt.Sprintf("image %d came back broken: %v", i+1, err))
+ continue
+ }
+ case d.URL != "":
+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+ img, mt, err = FetchImage(ctx, PublicClient(), d.URL)
+ cancel()
+ if err != nil {
+ lost = append(lost, fmt.Sprintf("image %d was drawn but couldn't be downloaded (%v); the vendor keeps it at %s for a while: download it from there", i+1, err, d.URL))
+ continue
+ }
+ default:
+ continue
+ }
+ // what the bytes are, over what the vendor says
+ if sniffed := http.DetectContentType(img); strings.HasPrefix(sniffed, "image/") {
+ mt = sniffed
+ }
+ if !strings.HasPrefix(mt, "image/") {
+ mt = "image/png"
+ }
+ urls = append(urls, base+ImageMCPFiles+s.imageMCPKeepFile(img, mt, key))
+ if len(img) <= imageMCPInline {
+ content = append(content, map[string]any{"type": "image", "data": base64.StdEncoding.EncodeToString(img), "mimeType": mt})
+ }
+ if d.Revised != "" {
+ revised = append(revised, d.Revised)
+ }
+ }
+ if len(urls) == 0 {
+ return toolResult(strings.Join(lost, "\n"), true)
+ }
+ var b strings.Builder
+ what := "The image is"
+ if len(urls) > 1 {
+ what = "The images are"
+ }
+ fmt.Fprintf(&b, "Generated with %s. %s in this answer and can be downloaded for an hour from (with the same Authorization header as this server, from another computer):\n", out.Model, what)
+ for _, u := range urls {
+ fmt.Fprintf(&b, "- %s\n", u)
+ }
+ if len(content) < len(urls) {
+ fmt.Fprintf(&b, "Images over %d MB are only at their URL.\n", imageMCPInline>>20)
+ }
+ for _, l := range lost {
+ fmt.Fprintf(&b, "Not saved: %s\n", l)
+ }
+ if len(revised) > 0 {
+ fmt.Fprintf(&b, "The model drew from this prompt: %s\n", revised[0])
+ }
+ if t := strings.TrimSpace(out.Text); t != "" {
+ fmt.Fprintf(&b, "The model said: %s\n", t)
+ }
+ content = append([]any{map[string]any{"type": "text", "text": b.String()}}, content...)
+ return map[string]any{"content": content, "isError": false}
+}
+
+// imageMCPFilm starts generate_video's video and answers with where to
+// download it, or, while it is still being made after imageMCPWithin,
+// with its id.
+func (s *Server) imageMCPFilm(r *http.Request, raw json.RawMessage) map[string]any {
+ if off, ok := imageGenOff(); ok {
+ return off
+ }
+ var a struct {
+ Prompt string `json:"prompt"`
+ Seconds any `json:"seconds"`
+ Size string `json:"size"`
+ Image string `json:"image"`
+ References []string `json:"reference_images"`
+ }
+ if err := json.Unmarshal(raw, &a); err != nil {
+ return toolResult("invalid arguments: "+err.Error(), true)
+ }
+ if strings.TrimSpace(a.Prompt) == "" {
+ return toolResult(GenerateVideoTool+" needs a prompt", true)
+ }
+ req := map[string]any{"prompt": a.Prompt}
+ switch v := a.Seconds.(type) {
+ case nil:
+ case float64:
+ if v != float64(int(v)) || v < 1 || v > 15 {
+ return toolResult("seconds is a whole number from 1 to 15", true)
+ }
+ req["seconds"] = strconv.Itoa(int(v))
+ case string:
+ n, err := strconv.Atoi(strings.TrimSpace(v))
+ if err != nil || n < 1 || n > 15 {
+ return toolResult("seconds is a whole number from 1 to 15", true)
+ }
+ req["seconds"] = strconv.Itoa(n)
+ default:
+ return toolResult("seconds is a whole number from 1 to 15", true)
+ }
+ if a.Size != "" {
+ req["size"] = a.Size
+ }
+ size := len(a.Prompt)
+ if a.Image != "" {
+ src, err := mcpImageSrc(r, strings.TrimSpace(a.Image), GenerateVideoTool)
+ if err != nil {
+ return toolResult(err.Error(), true)
+ }
+ size += len(src)
+ req["input_reference"] = map[string]string{"image_url": src}
+ }
+ if len(a.References) > 0 {
+ var refs []map[string]string
+ for _, ref := range a.References {
+ src, err := mcpImageSrc(r, strings.TrimSpace(ref), GenerateVideoTool)
+ if err != nil {
+ return toolResult(err.Error(), true)
+ }
+ size += len(src)
+ refs = append(refs, map[string]string{"image_url": src})
+ }
+ req["reference_images"] = refs
+ }
+ release, err := reserveFor(access.Caller(r.Context()), int64(size), "", limitClock())
+ if err != nil {
+ return toolResult(err.Error(), true)
+ }
+ body, _ := json.Marshal(req)
+ rec := &replyRecorder{header: http.Header{}}
+ s.videosCreate(rec, inProcess(r, r.Context(), http.MethodPost, "/v1/videos", body))
+ release()
+ if rec.code() != http.StatusOK {
+ return toolResult("no video: "+imageMCPSaid(rec.body.Bytes()), true)
+ }
+ var v struct {
+ ID string `json:"id"`
+ }
+ if json.Unmarshal(rec.body.Bytes(), &v) != nil || v.ID == "" {
+ return toolResult("no video: the answer has no id: "+imageMCPSaid(rec.body.Bytes()), true)
+ }
+ return s.imageMCPVideo(r, v.ID)
+}
+
+// imageMCPVideo asks after video id until it is made, fails, or
+// imageMCPWithin has passed.
+func (s *Server) imageMCPVideo(r *http.Request, id string) map[string]any {
+ deadline := time.Now().Add(imageMCPWithin)
+ started := time.Now()
+ for {
+ in := inProcess(r, r.Context(), http.MethodGet, "/v1/videos/"+id, nil)
+ in.SetPathValue("id", id)
+ rec := &replyRecorder{header: http.Header{}}
+ s.videosGet(rec, in)
+ if rec.code() != http.StatusOK {
+ return toolResult(fmt.Sprintf("asking after the video (%s): %s", id, imageMCPSaid(rec.body.Bytes())), true)
+ }
+ var v struct {
+ Status string `json:"status"`
+ Model string `json:"model"`
+ Seconds string `json:"seconds"`
+ Error *struct {
+ Message string `json:"message"`
+ } `json:"error"`
+ }
+ json.Unmarshal(rec.body.Bytes(), &v)
+ switch v.Status {
+ case "completed":
+ length := ""
+ if v.Seconds != "" {
+ length = v.Seconds + "-second "
+ }
+ return toolResult(fmt.Sprintf("Generated a %svideo with %s. Download the mp4 from (with the same Authorization header as this server, from another computer):\n- %s/v1/videos/%s/content\n", length, v.Model, imageMCPBase(r), id), false)
+ case "failed":
+ msg := "it gave no reason"
+ if v.Error != nil && v.Error.Message != "" {
+ msg = v.Error.Message
+ }
+ return toolResult("the video failed: "+msg, true)
+ }
+ if !time.Now().Add(imageMCPPoll).Before(deadline) {
+ return toolResult(fmt.Sprintf("The video is still being made (%s, %d s so far; it can take a few minutes). It goes on without this call. "+
+ "Don't ask for it again: call %s with id %q to wait for it and get where to download it.\n", cmp.Or(v.Status, "queued"), int(time.Since(started)/time.Second), GenerationResultTool, id), false)
+ }
+ select {
+ case <-r.Context().Done():
+ return toolResult("the call ended before the video was made: call "+GenerationResultTool+" with id "+strconv.Quote(id), true)
+ case <-time.After(imageMCPPoll):
+ }
+ }
+}
+
+// imageMCPResult is generation_result's answer for id: a video's, by its
+// own id, or a drawing's, started by the same key.
+func (s *Server) imageMCPResult(r *http.Request, id string) map[string]any {
+ if id == "" {
+ return toolResult(GenerationResultTool+" needs the id "+GenerateImageTool+" or "+GenerateVideoTool+" answered with", true)
+ }
+ if strings.HasPrefix(id, "video_") {
+ return s.imageMCPVideo(r, id)
+ }
+ st := &s.imageStore
+ st.mu.Lock()
+ j := st.jobs[id]
+ st.mu.Unlock()
+ if j == nil || j.key != access.Caller(r.Context()).KeyID {
+ return toolResult(fmt.Sprintf("no image is being made as %q (magpie may have been restarted, or it was made over an hour ago): generate it again", id), true)
+ }
+ return s.imageMCPAwait(r, j)
+}
+
+// imageMCPStart runs draw as a job started by key.
+func (s *Server) imageMCPStart(key string, draw func() map[string]any) *imageJob {
+ st := &s.imageStore
+ j := &imageJob{id: "img_" + randomHex(12), key: key, started: time.Now(), done: make(chan struct{})}
+ st.mu.Lock()
+ if st.jobs == nil {
+ st.jobs = map[string]*imageJob{}
+ }
+ for id, old := range st.jobs {
+ select {
+ case <-old.done:
+ if time.Since(old.started) > imageMCPKeep {
+ delete(st.jobs, id)
+ }
+ default:
+ }
+ }
+ st.jobs[j.id] = j
+ st.running.Add(1)
+ st.mu.Unlock()
+ go func() {
+ defer st.running.Done()
+ defer close(j.done)
+ j.result = draw()
+ }()
+ return j
+}
+
+// imageMCPAwait waits up to imageMCPWithin for j.
+func (s *Server) imageMCPAwait(r *http.Request, j *imageJob) map[string]any {
+ limit := time.NewTimer(imageMCPWithin)
+ defer limit.Stop()
+ select {
+ case <-j.done:
+ return j.result
+ case <-limit.C:
+ case <-r.Context().Done():
+ }
+ return toolResult(fmt.Sprintf("The image is still being made (%d s so far; it can take a few minutes). It goes on without this call. "+
+ "Don't ask for it again: call %s with id %q to wait for it and get it.\n", int(time.Since(j.started)/time.Second), GenerationResultTool, j.id), false)
+}
+
+// imageMCPKeepFile keeps img for an hour, for key, and is its file's name.
+func (s *Server) imageMCPKeepFile(img []byte, mt, key string) string {
+ st := &s.imageStore
+ name := randomHex(16) + extOf(mt)
+ st.mu.Lock()
+ defer st.mu.Unlock()
+ if st.files == nil {
+ st.files = map[string]*imageFile{}
+ }
+ st.files[name] = &imageFile{data: img, mime: mt, key: key, made: time.Now()}
+ st.order = append(st.order, name)
+ st.bytes += len(img)
+ for len(st.order) > 1 && (st.bytes > imageMCPFilesMost || time.Since(st.files[st.order[0]].made) > imageMCPKeep) {
+ old := st.order[0]
+ st.order = st.order[1:]
+ st.bytes -= len(st.files[old].data)
+ delete(st.files, old)
+ }
+ return name
+}
+
+// imageMCPFile serves an image the server made, to the key that had it
+// made, guarded as the server is.
+func (s *Server) imageMCPFile(w http.ResponseWriter, r *http.Request) {
+ if !local(r) && !sharedWith(r) {
+ http.Error(w, "magpie's images are served to another machine only when magpie is shared on the local network and the request carries a gateway key (Authorization: Bearer <key>)", http.StatusForbidden)
+ return
+ }
+ if o := r.Header.Get("Origin"); o != "" && !corsAllowed(o) {
+ http.Error(w, "magpie's images are served to agents, not web pages", http.StatusForbidden)
+ return
+ }
+ if r.Method != http.MethodGet && r.Method != http.MethodHead {
+ w.Header().Set("Allow", "GET, HEAD")
+ http.Error(w, "GET an image here", http.StatusMethodNotAllowed)
+ return
+ }
+ st := &s.imageStore
+ st.mu.Lock()
+ f := st.files[r.PathValue("name")]
+ if f != nil && time.Since(f.made) > imageMCPKeep {
+ f = nil
+ }
+ st.mu.Unlock()
+ // another key's image is as good as none: its name says nothing
+ if f == nil || f.key != access.Caller(r.Context()).KeyID {
+ http.Error(w, "no such image: images are kept for an hour, and only for the key that had them made", http.StatusNotFound)
+ return
+ }
+ w.Header().Set("Content-Type", f.mime)
+ w.Header().Set("X-Content-Type-Options", "nosniff")
+ w.Header().Set("Cache-Control", "private, max-age=3600")
+ w.Header().Set("Content-Length", strconv.Itoa(len(f.data)))
+ if r.Method == http.MethodHead {
+ return
+ }
+ w.Write(f.data)
+}
+
+// imageMCPBase is the address the agent reaches the gateway at, for the
+// URLs it is given.
+func imageMCPBase(r *http.Request) string {
+ if u := publicURL(); u != "" {
+ return u
+ }
+ scheme := "http"
+ if r.TLS != nil {
+ scheme = "https"
+ }
+ return scheme + "://" + r.Host
+}
+
+// inProcess is r, an MCP request, as a request of the gateway's own API at
+// path with body, carrying r's caller (its key, its agent) under ctx.
+func inProcess(r *http.Request, ctx context.Context, method, path string, body []byte) *http.Request {
+ in := r.Clone(ctx)
+ in.Method = method
+ u := *r.URL
+ u.Path, u.RawPath, u.RawQuery = path, "", ""
+ in.URL = &u
+ in.RequestURI = path
+ in.Body, in.ContentLength = http.NoBody, 0
+ if body != nil {
+ in.Body, in.ContentLength = io.NopCloser(bytes.NewReader(body)), int64(len(body))
+ in.Header.Set("Content-Type", "application/json")
+ } else {
+ in.Header.Del("Content-Type")
+ }
+ in.Header.Del("Content-Encoding")
+ in.Header.Del("Content-Length")
+ // a reply that comes back compressed couldn't be read here
+ in.Header.Del("Accept-Encoding")
+ in.Header.Del("Mcp-Session-Id")
+ return in
+}
+
+// replyRecorder takes what one of the gateway's handlers answers.
+type replyRecorder struct {
+ header http.Header
+ status int
+ body bytes.Buffer
+}
+
+func (w *replyRecorder) Header() http.Header { return w.header }
+
+func (w *replyRecorder) WriteHeader(code int) {
+ if w.status == 0 {
+ w.status = code
+ }
+}
+
+func (w *replyRecorder) Write(b []byte) (int, error) {
+ if w.status == 0 {
+ w.status = http.StatusOK
+ }
+ if w.body.Len()+len(b) > 512<<20 {
+ return 0, errors.New("the reply is too large")
+ }
+ return w.body.Write(b)
+}
+
+func (w *replyRecorder) code() int {
+ if w.status == 0 {
+ return http.StatusOK
+ }
+ return w.status
+}
+
+// imageMCPSaid is the message of an error the gateway answered with, else the
+// answer itself.
+func imageMCPSaid(data []byte) string {
+ var e struct {
+ Error *struct {
+ Message string `json:"message"`
+ } `json:"error"`
+ }
+ if json.Unmarshal(data, &e) == nil && e.Error != nil && e.Error.Message != "" {
+ return e.Error.Message
+ }
+ s := strings.TrimSpace(string(data))
+ if len(s) > 500 {
+ s = s[:500] + "…"
+ }
+ return s
+}
+
+func randomHex(n int) string {
+ b := make([]byte, n)
+ rand.Read(b)
+ return hex.EncodeToString(b)
+}
diff --git a/internal/gateway/image_mcp_test.go b/internal/gateway/image_mcp_test.go
new file mode 100644
index 000000000..597b15748
--- /dev/null
+++ b/internal/gateway/image_mcp_test.go
@@ -0,0 +1,381 @@
+package gateway
+
+import (
+ "encoding/base64"
+ "encoding/json"
+ "net/http"
+ "net/http/httptest"
+ "net/url"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/yetone/magpie/internal/access"
+ "github.com/yetone/magpie/internal/budget"
+ "github.com/yetone/magpie/internal/provider"
+ "github.com/yetone/magpie/internal/settings"
+)
+
+func imagePost(h http.Handler, addr, body string, hdr ...string) *httptest.ResponseRecorder {
+ r := httptest.NewRequest(http.MethodPost, ImageMCPPath, strings.NewReader(body))
+ r.RemoteAddr = addr
+ r.Host = "127.0.0.1:3425"
+ r.Header.Set("Content-Type", "application/json")
+ r.Header.Set("Accept", "application/json, text/event-stream")
+ for i := 0; i+1 < len(hdr); i += 2 {
+ r.Header.Set(hdr[i], hdr[i+1])
+ }
+ w := httptest.NewRecorder()
+ h.ServeHTTP(w, r)
+ return w
+}
+
+func imageGet(h http.Handler, addr, rawURL string, hdr ...string) *httptest.ResponseRecorder {
+ u, _ := url.Parse(rawURL)
+ r := httptest.NewRequest(http.MethodGet, u.RequestURI(), nil)
+ r.RemoteAddr = addr
+ r.Host = "127.0.0.1:3425"
+ for i := 0; i+1 < len(hdr); i += 2 {
+ r.Header.Set(hdr[i], hdr[i+1])
+ }
+ w := httptest.NewRecorder()
+ h.ServeHTTP(w, r)
+ return w
+}
+
+func imageToolCall(name string, args map[string]any) string {
+ b, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 5, "method": "tools/call", "params": map[string]any{"name": name, "arguments": args}})
+ return string(b)
+}
+
+type imageReply struct {
+ Result struct {
+ Content []struct {
+ Type, Text, Data, MimeType string
+ }
+ IsError bool
+ }
+ Error *struct{ Message string }
+}
+
+func readImageReply(t *testing.T, w *httptest.ResponseRecorder) imageReply {
+ t.Helper()
+ if w.Code != 200 {
+ t.Fatalf("%d %s", w.Code, w.Body.String())
+ }
+ var m imageReply
+ if err := json.Unmarshal(w.Body.Bytes(), &m); err != nil || len(m.Result.Content) == 0 {
+ t.Fatalf("%v: %s", err, w.Body.String())
+ }
+ return m
+}
+
+// the URLs a tool's text names, one a line ("- <url>")
+func urlsIn(text string) []string {
+ var out []string
+ for _, l := range strings.Split(text, "\n") {
+ if u, ok := strings.CutPrefix(l, "- "); ok && strings.Contains(u, "://") {
+ out = append(out, u)
+ }
+ }
+ return out
+}
+
+// magpie's image generation is an MCP server on the gateway (JonahFu on
+// X), for an agent that has no Magpie Image, in a gateway in Docker or on
+// another computer: it lists Magpie Image's tools with their arguments but
+// where to save, draws with the model Settings picks, and answers with the
+// image itself and a URL it can be downloaded from.
+func TestImageMCPDrawsAndServesTheImage(t *testing.T) {
+ s, e := easeled(t)
+ t.Setenv("MAGPIE_PUBLIC_URL", "")
+ t.Cleanup(s.imageStore.running.Wait)
+ h := lanGuard(s.Handler())
+ init := readMCP(t, imagePost(h, "127.0.0.1:5000", `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18"}}`))
+ if init.Result.ServerInfo.Name != "magpie-image" || init.Result.ProtocolVersion != "2025-06-18" {
+ t.Fatalf("initialize: %+v", init)
+ }
+ var list struct {
+ Result struct {
+ Tools []struct {
+ Name string
+ InputSchema struct {
+ Properties map[string]any
+ Required []string
+ }
+ }
+ }
+ }
+ w := imagePost(h, "127.0.0.1:5000", `{"jsonrpc":"2.0","id":2,"method":"tools/list"}`)
+ json.Unmarshal(w.Body.Bytes(), &list)
+ var names []string
+ for _, tl := range list.Result.Tools {
+ names = append(names, tl.Name)
+ if _, ok := tl.InputSchema.Properties["path"]; ok {
+ t.Fatalf("%s takes a path the gateway can't write to", tl.Name)
+ }
+ }
+ if strings.Join(names, ",") != "generate_image,generate_video,generation_result" || len(list.Result.Tools[0].InputSchema.Properties) != len(ImageToolProps("")) {
+ t.Fatalf("tools: %s", w.Body.String())
+ }
+ m := readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(GenerateImageTool, map[string]any{"prompt": "a magpie", "size": "1024x1024"})))
+ if m.Result.IsError || len(m.Result.Content) != 2 {
+ t.Fatalf("drawn: %+v", m)
+ }
+ img := m.Result.Content[1]
+ if img.Type != "image" || img.MimeType != "image/png" || img.Data != base64.StdEncoding.EncodeToString(pngBytes) {
+ t.Fatalf("image content %+v", img)
+ }
+ sent := e.got("/v1/images/generations")
+ if len(sent) != 1 || !strings.Contains(sent[0], `"prompt":"a magpie"`) || !strings.Contains(sent[0], `"size":"1024x1024"`) {
+ t.Fatalf("vendor was sent %v", sent)
+ }
+ urls := urlsIn(m.Result.Content[0].Text)
+ if len(urls) != 1 || !strings.HasPrefix(urls[0], "http://127.0.0.1:3425"+ImageMCPFiles) || !strings.HasSuffix(urls[0], ".png") || !strings.Contains(m.Result.Content[0].Text, "art/") {
+ t.Fatalf("text: %q", m.Result.Content[0].Text)
+ }
+ got := imageGet(h, "127.0.0.1:5000", urls[0])
+ if got.Code != 200 || got.Body.String() != string(pngBytes) || got.Header().Get("Content-Type") != "image/png" || got.Header().Get("X-Content-Type-Options") != "nosniff" {
+ t.Fatalf("file: %d %v %q", got.Code, got.Header(), got.Body.String())
+ }
+ if w := imageGet(h, "127.0.0.1:5000", "http://127.0.0.1:3425"+ImageMCPFiles+"nothing.png"); w.Code != 404 {
+ t.Fatalf("no such file: %d", w.Code)
+ }
+ // MAGPIE_PUBLIC_URL is the address the agent is given
+ t.Setenv("MAGPIE_PUBLIC_URL", "https://magpie.example.net/")
+ m = readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(GenerateImageTool, map[string]any{"prompt": "a magpie"})))
+ if u := urlsIn(m.Result.Content[0].Text); len(u) != 1 || !strings.HasPrefix(u[0], "https://magpie.example.net"+ImageMCPFiles) {
+ t.Fatalf("public URL: %q", m.Result.Content[0].Text)
+ }
+ // a reference image makes it an edit
+ m = readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(GenerateImageTool, map[string]any{"prompt": "bluer", "reference_images": []string{"data:image/png;base64," + base64.StdEncoding.EncodeToString(pngBytes)}})))
+ if m.Result.IsError || len(e.got("/v1/images/edits")) != 1 {
+ t.Fatalf("edit: %+v %v", m, e.got("/v1/images/edits"))
+ }
+}
+
+// Off in Settings, the tools say so, and nothing is drawn.
+func TestImageMCPSaysWhenOff(t *testing.T) {
+ s, e := easeled(t)
+ st := settings.Load()
+ st.ImageGen = "off"
+ if err := settings.Save(st); err != nil {
+ t.Fatal(err)
+ }
+ h := lanGuard(s.Handler())
+ for _, tool := range []string{GenerateImageTool, GenerateVideoTool} {
+ m := readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(tool, map[string]any{"prompt": "a magpie"})))
+ if !m.Result.IsError || !strings.Contains(m.Result.Content[0].Text, "Image generation is off in magpie") {
+ t.Fatalf("%s: %+v", tool, m)
+ }
+ }
+ if n := len(e.got("/v1/images/generations")); n != 0 {
+ t.Fatalf("drew %d times while off", n)
+ }
+}
+
+// The server is guarded as the vision server is: another computer only on
+// a shared magpie with a key, no web page, no file path from elsewhere;
+// and an image it made is served only to the key that had it made.
+func TestImageMCPGuarded(t *testing.T) {
+ s, e := easeled(t)
+ t.Setenv("MAGPIE_ADDR", "")
+ t.Setenv("MAGPIE_PUBLIC_URL", "")
+ t.Cleanup(s.imageStore.running.Wait)
+ h := lanGuard(s.Handler())
+ file, _ := pngFile(t)
+ const there = "192.0.2.7:4000"
+ draw := imageToolCall(GenerateImageTool, map[string]any{"prompt": "a magpie"})
+ if w := imagePost(h, there, draw); w.Code != http.StatusForbidden {
+ t.Fatalf("not shared: %d %s", w.Code, w.Body.String())
+ }
+ _, secrets := newCaller(t, "Laptop", "Phone")
+ if w := imagePost(h, there, draw); w.Code != http.StatusUnauthorized {
+ t.Fatalf("shared, no key: %d %s", w.Code, w.Body.String())
+ }
+ if w := imagePost(h, "127.0.0.1:50000", draw, "Origin", "http://localhost:5173"); w.Code != http.StatusForbidden {
+ t.Fatalf("a page on this computer: %d %s", w.Code, w.Body.String())
+ }
+ for _, tool := range []string{GenerateImageTool, GenerateVideoTool} {
+ args := map[string]any{"prompt": "a magpie", "reference_images": []string{file}}
+ m := readImageReply(t, imagePost(h, there, imageToolCall(tool, args), "Authorization", "Bearer "+secrets[0]))
+ if !m.Result.IsError || !strings.Contains(m.Result.Content[0].Text, tool+" reads a file only for an agent on the computer magpie runs on") {
+ t.Fatalf("%s, another computer's file path: %+v", tool, m)
+ }
+ }
+ if n := len(e.got("/v1/images/edits")); n != 0 {
+ t.Fatalf("drew from another computer's path %d times", n)
+ }
+ m := readImageReply(t, imagePost(h, there, draw, "Authorization", "Bearer "+secrets[0]))
+ urls := urlsIn(m.Result.Content[0].Text)
+ if m.Result.IsError || len(urls) != 1 {
+ t.Fatalf("shared, with its key: %+v", m)
+ }
+ if w := imageGet(h, there, urls[0], "Authorization", "Bearer "+secrets[0]); w.Code != 200 || w.Body.String() != string(pngBytes) {
+ t.Fatalf("its own key: %d %s", w.Code, w.Body.String())
+ }
+ if w := imageGet(h, there, urls[0]); w.Code != http.StatusUnauthorized {
+ t.Fatalf("no key: %d %s", w.Code, w.Body.String())
+ }
+ if w := imageGet(h, there, urls[0], "Authorization", "Bearer "+secrets[1]); w.Code != http.StatusNotFound {
+ t.Fatalf("another key: %d %s", w.Code, w.Body.String())
+ }
+ if w := imageGet(h, "127.0.0.1:5000", urls[0]); w.Code != http.StatusNotFound {
+ t.Fatalf("keyless on this computer: %d %s", w.Code, w.Body.String())
+ }
+ if w := imageGet(h, "127.0.0.1:5000", urls[0], "Authorization", "Bearer "+secrets[0], "Origin", "http://localhost:5173"); w.Code != http.StatusForbidden {
+ t.Fatalf("a page: %d %s", w.Code, w.Body.String())
+ }
+}
+
+// A gateway key past its limit draws nothing through the server.
+func TestImageMCPHeldToKeyLimit(t *testing.T) {
+ s, e := easeled(t)
+ t.Cleanup(s.imageStore.running.Wait)
+ budget.Forget()
+ pinLimitClock(t)
+ calls := limitedUpstream(t, 0)
+ keys, secrets := newCaller(t, "Capped")
+ setLimit(t, keys[0].ID, &access.Limit{Period: "day", Tokens: 700})
+ h := lanGuard(s.Handler())
+ for range 2 {
+ r := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(chatReq))
+ r.RemoteAddr = "127.0.0.1:5000"
+ r.Header.Set("Authorization", "Bearer "+secrets[0])
+ w := httptest.NewRecorder()
+ h.ServeHTTP(w, r)
+ if w.Code != 200 {
+ t.Fatal(w.Code, w.Body.String())
+ }
+ }
+ if calls.Load() != 2 {
+ t.Fatal(calls.Load())
+ }
+ for _, tool := range []string{GenerateImageTool, GenerateVideoTool} {
+ m := readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(tool, map[string]any{"prompt": "a magpie"}), "Authorization", "Bearer "+secrets[0]))
+ if !m.Result.IsError || !strings.Contains(m.Result.Content[0].Text, `"Capped"`) {
+ t.Fatalf("%s, a spent key: %+v", tool, m)
+ }
+ }
+ if n := len(e.got("/v1/images/generations")); n != 0 {
+ t.Fatalf("a spent key drew %d times", n)
+ }
+ // without a key, from this computer, it draws
+ if m := readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(GenerateImageTool, map[string]any{"prompt": "a magpie"}))); m.Result.IsError {
+ t.Fatalf("keyless: %+v", m)
+ }
+}
+
+// A drawing that outlasts the call goes on, and generation_result gets it
+// by the id the call answered with; for the key that started it only.
+func TestImageMCPSlowDrawingIsWaitedOn(t *testing.T) {
+ fresh(t)
+ t.Setenv("MAGPIE_PUBLIC_URL", "")
+ gate := make(chan struct{})
+ e := &easel{}
+ up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ <-gate
+ e.ServeHTTP(w, r)
+ }))
+ t.Cleanup(up.Close)
+ if err := provider.Save(provider.Provider{ID: "art", Name: "Art", Chat: up.URL + "/v1", Key: "key", Models: []string{"gpt-image-1"}}); err != nil {
+ t.Fatal(err)
+ }
+ s := New()
+ opened := false
+ open := func() {
+ if !opened {
+ opened = true
+ close(gate)
+ }
+ }
+ // the job ends once the vendor answers
+ t.Cleanup(s.imageStore.running.Wait)
+ t.Cleanup(open)
+ h := lanGuard(s.Handler())
+ _, secrets := newCaller(t, "Mine", "Theirs")
+ was := imageMCPWithin
+ t.Cleanup(func() { imageMCPWithin = was })
+ imageMCPWithin = 10 * time.Millisecond
+ m := readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(GenerateImageTool, map[string]any{"prompt": "a magpie"}), "Authorization", "Bearer "+secrets[0]))
+ text := m.Result.Content[0].Text
+ i := strings.Index(text, `"img_`)
+ if m.Result.IsError || i < 0 || !strings.Contains(text, GenerationResultTool) {
+ t.Fatalf("still drawing: %+v", m)
+ }
+ id := strings.Trim(strings.Fields(text[i:])[0], `".,`)
+ // another key's call can't have it
+ m = readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(GenerationResultTool, map[string]any{"id": id}), "Authorization", "Bearer "+secrets[1]))
+ if !m.Result.IsError || !strings.Contains(m.Result.Content[0].Text, "no image is being made") {
+ t.Fatalf("another key: %+v", m)
+ }
+ open()
+ imageMCPWithin = time.Minute
+ m = readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(GenerationResultTool, map[string]any{"id": id}), "Authorization", "Bearer "+secrets[0]))
+ if m.Result.IsError || len(m.Result.Content) != 2 || m.Result.Content[1].Data != base64.StdEncoding.EncodeToString(pngBytes) {
+ t.Fatalf("waited on: %+v", m)
+ }
+ if n := len(e.got("/v1/images/generations")); n != 1 {
+ t.Fatalf("drew %d times", n)
+ }
+}
+
+// generate_video starts a video with the model that makes them, waits on
+// it, and answers with the gateway's URL to download it from; one that
+// outlasts the call is waited on by its id.
+func TestImageMCPMakesVideos(t *testing.T) {
+ grokSignedIn(t)
+ newGrokMedia(t)
+ t.Setenv("MAGPIE_PUBLIC_URL", "")
+ s := New()
+ h := lanGuard(s.Handler())
+ was, wasPoll := imageMCPWithin, imageMCPPoll
+ t.Cleanup(func() { imageMCPWithin, imageMCPPoll = was, wasPoll })
+ // still being made when the call answers: its id is waited on
+ imageMCPWithin, imageMCPPoll = 0, time.Millisecond
+ m := readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(GenerateVideoTool, map[string]any{"prompt": "a kite sways", "seconds": 6})))
+ text := m.Result.Content[0].Text
+ i := strings.Index(text, `"video_`)
+ if m.Result.IsError || i < 0 || !strings.Contains(text, GenerationResultTool) {
+ t.Fatalf("still making: %+v", m)
+ }
+ id := strings.Trim(strings.Fields(text[i:])[0], `".,`)
+ imageMCPWithin = time.Minute
+ m = readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(GenerationResultTool, map[string]any{"id": id})))
+ text = m.Result.Content[0].Text
+ urls := urlsIn(text)
+ if m.Result.IsError || len(urls) != 1 || urls[0] != "http://127.0.0.1:3425/v1/videos/"+id+"/content" || !strings.Contains(text, "6-second") || !strings.Contains(text, "grok/grok-imagine-video") {
+ t.Fatalf("video: %+v", m)
+ }
+ if w := imageGet(h, "127.0.0.1:5000", urls[0]); w.Code != 200 || w.Body.String() != "fake-mp4-bytes" {
+ t.Fatalf("content: %d %s", w.Code, w.Body.String())
+ }
+ // made within the call: its URL at once
+ m = readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(GenerateVideoTool, map[string]any{"prompt": "a kite"})))
+ if u := urlsIn(m.Result.Content[0].Text); m.Result.IsError || len(u) != 1 || !strings.HasSuffix(u[0], "/content") {
+ t.Fatalf("at once: %+v", m)
+ }
+ if m := readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(GenerateVideoTool, map[string]any{"prompt": "a kite", "seconds": 40}))); !m.Result.IsError || !strings.Contains(m.Result.Content[0].Text, "1 to 15") {
+ t.Fatalf("seconds: %+v", m)
+ }
+ // the vendor's video failing is said
+ m = readImageReply(t, imagePost(h, "127.0.0.1:5000", imageToolCall(GenerateVideoTool, map[string]any{"prompt": "FAILING"})))
+ if !m.Result.IsError || !strings.Contains(m.Result.Content[0].Text, "the video failed: image_url must") {
+ t.Fatalf("failed: %+v", m)
+ }
+}
+
+// The gateway serves the server (and its files) on the address `magpie
+// web` and a shared magpie forward.
+func TestImageMCPIsServed(t *testing.T) {
+ for _, p := range []string{ImageMCPPath, ImageMCPFiles + "x.png"} {
+ if !ClientPath(p) {
+ t.Fatalf("%s isn't a client path", p)
+ }
+ }
+ fresh(t)
+ w := imageGet(New().Handler(), "127.0.0.1:5000", "http://127.0.0.1:3425/v1")
+ if w.Code != 200 || !strings.Contains(w.Body.String(), `"`+ImageMCPPath+`"`) {
+ t.Fatalf("apis: %d %s", w.Code, w.Body.String())
+ }
+}
diff --git a/internal/gateway/oneport_test.go b/internal/gateway/oneport_test.go
index 85c0b6a22..44f697e30 100644
--- a/internal/gateway/oneport_test.go
+++ b/internal/gateway/oneport_test.go
@@ -27,6 +27,7 @@ func TestClientPathCoversHandler(t *testing.T) {
"claudeMCPPath": claudeMCPPath, "CodexPath": CodexPath, "CodexCatalogPath": CodexCatalogPath,
"AlphaSearchPath": AlphaSearchPath, "HeapPath": HeapPath, "CPUPath": CPUPath,
"GoroutinesPath": GoroutinesPath, "SearchMCPPath": SearchMCPPath, "VisionMCPPath": VisionMCPPath,
+ "ImageMCPPath": ImageMCPPath, "ImageMCPFiles": ImageMCPFiles,
"ToolsMCPPath": ToolsMCPPath, "provider.RemoteCardsPath": provider.RemoteCardsPath,
"provider.RemoteRefreshPath": provider.RemoteRefreshPath,
}
diff --git a/internal/gateway/vision_mcp.go b/internal/gateway/vision_mcp.go
index d7e560456..3b4ce6a38 100644
--- a/internal/gateway/vision_mcp.go
+++ b/internal/gateway/vision_mcp.go
@@ -112,7 +112,7 @@ func (s *Server) visionMCPRun(r *http.Request, image, prompt string) (string, er
if image == "" {
return "", searchMCPError("analyze_image needs an image: a file path, an http(s) URL or a data URL")
}
- src, err := visionSrc(r, image)
+ src, err := mcpImageSrc(r, image, visionMCPTool)
if err != nil {
return "", err
}
@@ -144,16 +144,18 @@ func (s *Server) visionMCPRun(r *http.Request, image, prompt string) (string, er
return said, nil
}
-// visionSrc is the image as the describer is sent it: a URL or data URL as
-// it is, a file read into a data URL. A file is read only for an agent on
+// mcpImageSrc is an image an MCP tool (tool, which names it in what it says)
+// was given, as the gateway's own APIs take it: a URL or data URL as it
+// is, a file read into a data URL. A file is read only for an agent on
// this computer: another machine's path names none of its own here, and a
-// shared magpie's files aren't its callers'.
-func visionSrc(r *http.Request, image string) (string, error) {
+// shared magpie's files aren't its callers'. analyze_image and the image
+// server's tools (image_mcp.go) read their images through it.
+func mcpImageSrc(r *http.Request, image, tool string) (string, error) {
low := strings.ToLower(image)
switch {
case strings.HasPrefix(low, "data:"):
if !strings.HasPrefix(low, "data:image/") || !strings.Contains(low[:min(len(low), 64)], ";base64,") {
- return "", searchMCPError("analyze_image takes a data URL of an image in base64 (data:image/png;base64,...)")
+ return "", searchMCPError(tool + " takes a data URL of an image in base64 (data:image/png;base64,...)")
}
return image, nil
case strings.HasPrefix(low, "http://"), strings.HasPrefix(low, "https://"):
@@ -169,31 +171,31 @@ func visionSrc(r *http.Request, image string) (string, error) {
}
}
if !filepath.IsAbs(path) {
- return "", searchMCPError("analyze_image takes an absolute file path (the gateway doesn't know your agent's working folder), an http(s) URL or a data URL: " + image + " is none of them")
+ return "", searchMCPError(tool + " takes an absolute file path (the gateway doesn't know your agent's working folder), an http(s) URL or a data URL: " + image + " is none of them")
}
if !local(r) {
- return "", searchMCPError("analyze_image reads a file only for an agent on the computer magpie runs on: from another computer, send the image as a data URL (data:image/png;base64,...) or an http(s) URL")
+ return "", searchMCPError(tool + " reads a file only for an agent on the computer magpie runs on: from another computer, send the image as a data URL (data:image/png;base64,...) or an http(s) URL")
}
f, err := os.Open(path)
if err != nil {
- return "", searchMCPError("analyze_image couldn't read " + path + ": " + err.Error())
+ return "", searchMCPError(tool + " couldn't read " + path + ": " + err.Error())
}
defer f.Close()
if st, err := f.Stat(); err == nil && st.IsDir() {
return "", searchMCPError(path + " is a folder, not an image")
} else if err == nil && st.Size() > visionFileMax {
- return "", searchMCPError(fmt.Sprintf("%s is %d MB: analyze_image reads images up to %d MB", path, st.Size()>>20, visionFileMax>>20))
+ return "", searchMCPError(fmt.Sprintf("%s is %d MB: %s reads images up to %d MB", path, st.Size()>>20, tool, visionFileMax>>20))
}
b, err := io.ReadAll(io.LimitReader(f, visionFileMax+1))
if err != nil {
- return "", searchMCPError("analyze_image couldn't read " + path + ": " + err.Error())
+ return "", searchMCPError(tool + " couldn't read " + path + ": " + err.Error())
}
if len(b) > visionFileMax {
- return "", searchMCPError(fmt.Sprintf("%s is over %d MB: analyze_image reads images up to %d MB", path, visionFileMax>>20, visionFileMax>>20))
+ return "", searchMCPError(fmt.Sprintf("%s is over %d MB: %s reads images up to %d MB", path, visionFileMax>>20, tool, visionFileMax>>20))
}
mime := http.DetectContentType(b)
if !strings.HasPrefix(mime, "image/") {
- return "", searchMCPError(path + " isn't an image analyze_image can read (PNG, JPEG, GIF or WebP): it reads as " + mime)
+ return "", searchMCPError(path + " isn't an image " + tool + " can read (PNG, JPEG, GIF or WebP): it reads as " + mime)
}
return "data:" + mime + ";base64," + base64.StdEncoding.EncodeToString(b), nil
}
diff --git a/internal/gui/assets/app.js b/internal/gui/assets/app.js
index 217843abc..ef4a12c58 100644
--- a/internal/gui/assets/app.js
+++ b/internal/gui/assets/app.js
@@ -21230,17 +21230,17 @@ function renderImages(s, keep) {
r.append(who, val);
box.append(r);
renderImageGen(s, keep, box);
- // the same image recognition as an MCP server of the gateway's (JonahFu
- // on X), for an agent to mount as its analyze_image tool
- const mcpURL = mcpAddress(s, "/mcp/magpie/vision");
- const mcp = el("div", "row pref");
- const mwho = el("div", "who");
- mwho.append(el("div", "name", t("Image recognition MCP server")), el("div", "sub", t("Agents can add image recognition as an MCP server (Streamable HTTP) with one analyze_image tool, which reads an image with the Image recognition model from a file path, a URL or a data URL. From another computer, send a gateway key as Authorization: Bearer <key> and the image as a URL or data URL.")));
- const mval = el("div", "val");
- mval.append(el("code", "", mcpURL), copyBtn(mcpURL, t("Address")));
- mcp.append(mwho, mval);
- mcp.id = "visionMcpRow";
- box.append(mcp);
+ // both as the gateway's MCP servers, for an agent with no Library (JonahFu on X)
+ const mcpRow = (id, path, name, sub) => {
+ const url = mcpAddress(s, path), r = el("div", "row pref"), who = el("div", "who"), val = el("div", "val");
+ who.append(el("div", "name", name), el("div", "sub", sub));
+ val.append(el("code", "", url), copyBtn(url, t("Address")));
+ r.append(who, val);
+ r.id = id;
+ box.append(r);
+ };
+ mcpRow("visionMcpRow", "/mcp/magpie/vision", t("Image recognition MCP server"), t("Agents can add image recognition as an MCP server (Streamable HTTP) with one analyze_image tool, which reads an image with the Image recognition model from a file path, a URL or a data URL. From another computer, send a gateway key as Authorization: Bearer <key> and the image as a URL or data URL."));
+ mcpRow("imageMcpRow", "/mcp/magpie/image", t("Image generation MCP server"), t("Agents can add image generation as an MCP server (Streamable HTTP) with generate_image, generate_video and generation_result tools, which draw with the Image generation model. An image comes back in the answer and at a URL kept for an hour, a video at a URL; nothing is saved in the agent's project. From another computer, send a gateway key as Authorization: Bearer <key>."));
}
// renderImageGen: the model magpie's generate_image tool draws with — the
@@ -21269,7 +21269,7 @@ function renderImageGen(s, keep, box) {
const libOff = featuresOff.has("library");
const sub = el("div", "sub wraps",
v === "off" ? t("Agents given Magpie Image can't generate images or videos: the tool says it is off")
- : gatewayMode ? t("The model Magpie Image draws with, for the agents given the Magpie Image tool; images are saved in their project")
+ : gatewayMode ? t("The model the Image generation MCP server below draws with, for the agents that add it; images come back in their answer")
: libOff ? t("The model Magpie Image draws with. Agents get the Magpie Image tool from the Library, which is turned off in Plugins › Built-in; images are saved in their project")
: t("The model Magpie Image draws with. Agents get the Magpie Image tool from the Library; images are saved in their project"));
who.append(el("div", "name", t("Image generation")), sub);
diff --git a/internal/gui/assets/i18n.js b/internal/gui/assets/i18n.js
index 55cdf041c..59af24716 100644
--- a/internal/gui/assets/i18n.js
+++ b/internal/gui/assets/i18n.js
@@ -904,7 +904,9 @@ const I18N = {
"How long a routing group's classifier is given to say which intent a turn is. The turn waits for it before it goes anywhere, and past it goes on without its answer. A slow classifier may need longer": "路由组的意图判断模型判断这一轮属于哪种意图可用的时间。这一轮要等它判断完才会发出,超过则不等它的回答直接继续。较慢的判断模型可能需要更长时间",
"Image generation timeout": "生图超时",
"How long a provider is given to draw an image and send it, before the request is given up. A slow image endpoint may need longer": "提供商生成并返回一张图可用的时间,超过即放弃这次请求。较慢的生图接口可能需要更长",
- "The model Magpie Image draws with, for the agents given the Magpie Image tool; images are saved in their project": "Magpie Image 生图所用的模型,供添加了 Magpie Image 工具的 Agent 使用;图片保存在其项目目录",
+ "The model the Image generation MCP server below draws with, for the agents that add it; images come back in their answer": "下方生图 MCP 服务器所用的模型,供添加了它的智能体使用;图片随回答返回",
+ "Image generation MCP server": "生图 MCP 服务器",
+ "Agents can add image generation as an MCP server (Streamable HTTP) with generate_image, generate_video and generation_result tools, which draw with the Image generation model. An image comes back in the answer and at a URL kept for an hour, a video at a URL; nothing is saved in the agent's project. From another computer, send a gateway key as Authorization: Bearer <key>.": "智能体可以把生图添加为 MCP 服务器(Streamable HTTP),提供 generate_image、generate_video 和 generation_result 工具,用生图模型生成。图片随回答返回,并可在保留一小时的 URL 下载;视频以 URL 返回;不会保存到智能体的项目中。从其他电脑访问时,请以 Authorization: Bearer <key> 发送网关密钥。",
"The model Magpie Image draws with. Agents get the Magpie Image tool from the Library, which is turned off in Plugins › Built-in; images are saved in their project": "Magpie Image 生图所用的模型。Agent 要从资源库添加 Magpie Image 工具,而资源库在「插件 › 内置」中已关闭;图片保存在其项目目录",
"The model Magpie Image draws with. Agents get the Magpie Image tool from the Library; images are saved in their project": "Magpie Image 生图所用的模型。Agent 要从资源库添加 Magpie Image 工具才能生图;图片保存在其项目目录",
"Turn on the Library": "启用资源库",
@@ -4866,7 +4868,9 @@ const I18N = {
"How long a routing group's classifier is given to say which intent a turn is. The turn waits for it before it goes anywhere, and past it goes on without its answer. A slow classifier may need longer": "路由組的意圖判斷模型判斷這一輪屬於哪種意圖可用的時間。這一輪要等它判斷完才會送出,超過則不等它的回答直接繼續。較慢的判斷模型可能需要更長時間",
"Image generation timeout": "生圖逾時",
"How long a provider is given to draw an image and send it, before the request is given up. A slow image endpoint may need longer": "提供者生成並傳回一張圖可用的時間,超過即放棄這次請求。較慢的生圖端點可能需要更長",
- "The model Magpie Image draws with, for the agents given the Magpie Image tool; images are saved in their project": "Magpie Image 生圖所用的模型,供新增了 Magpie Image 工具的 Agent 使用;圖片儲存在其專案目錄",
+ "The model the Image generation MCP server below draws with, for the agents that add it; images come back in their answer": "下方生圖 MCP 伺服器所用的模型,供新增了它的智慧體使用;圖片隨回答傳回",
+ "Image generation MCP server": "生圖 MCP 伺服器",
+ "Agents can add image generation as an MCP server (Streamable HTTP) with generate_image, generate_video and generation_result tools, which draw with the Image generation model. An image comes back in the answer and at a URL kept for an hour, a video at a URL; nothing is saved in the agent's project. From another computer, send a gateway key as Authorization: Bearer <key>.": "智慧體可以把生圖新增為 MCP 伺服器(Streamable HTTP),提供 generate_image、generate_video 和 generation_result 工具,用生圖模型生成。圖片隨回答傳回,並可在保留一小時的 URL 下載;影片以 URL 傳回;不會儲存到智慧體的專案中。從其他電腦存取時,請以 Authorization: Bearer <key> 傳送閘道金鑰。",
"The model Magpie Image draws with. Agents get the Magpie Image tool from the Library, which is turned off in Plugins › Built-in; images are saved in their project": "Magpie Image 生圖所用的模型。Agent 要從資源庫新增 Magpie Image 工具,而資源庫在「外掛 › 內建」中已關閉;圖片儲存在其專案目錄",
"The model Magpie Image draws with. Agents get the Magpie Image tool from the Library; images are saved in their project": "Magpie Image 生圖所用的模型。Agent 要從資源庫新增 Magpie Image 工具才能生圖;圖片儲存在其專案目錄",
"Turn on the Library": "啟用資源庫",
@@ -8800,7 +8804,9 @@ const I18N = {
"How long a routing group's classifier is given to say which intent a turn is. The turn waits for it before it goes anywhere, and past it goes on without its answer. A slow classifier may need longer": "ルーティンググループの分類モデルが、このターンがどのインテントかを答えるまで待つ時間。ターンは答えを待ってから送られ、過ぎると答えなしで進みます。遅い分類モデルではもっと長く必要なことがあります",
"Image generation timeout": "画像生成のタイムアウト",
"How long a provider is given to draw an image and send it, before the request is given up. A slow image endpoint may need longer": "プロバイダーが画像を生成して送るまで待つ時間。過ぎるとリクエストは打ち切られます。遅い画像エンドポイントにはより長い時間が必要なことがあります",
- "The model Magpie Image draws with, for the agents given the Magpie Image tool; images are saved in their project": "Magpie Image ツールを付与したエージェントが画像生成に使うモデルです。画像はそのプロジェクトに保存されます",
+ "The model the Image generation MCP server below draws with, for the agents that add it; images come back in their answer": "下の画像生成 MCP サーバーが使うモデルです(追加したエージェント向け)。画像は回答で返されます",
+ "Image generation MCP server": "画像生成 MCP サーバー",
+ "Agents can add image generation as an MCP server (Streamable HTTP) with generate_image, generate_video and generation_result tools, which draw with the Image generation model. An image comes back in the answer and at a URL kept for an hour, a video at a URL; nothing is saved in the agent's project. From another computer, send a gateway key as Authorization: Bearer <key>.": "エージェントは、画像生成を generate_image、generate_video、generation_result ツールを持つ MCP サーバー(Streamable HTTP)として追加できます。画像生成のモデルで生成します。画像は回答に含まれ、1 時間保持される URL からもダウンロードできます。動画は URL で返されます。エージェントのプロジェクトには何も保存されません。別のコンピューターからは、ゲートウェイキーを Authorization: Bearer <key> として送ってください。",
"The model Magpie Image draws with. Agents get the Magpie Image tool from the Library, which is turned off in Plugins › Built-in; images are saved in their project": "Magpie Image が画像生成に使うモデルです。エージェントにはライブラリから Magpie Image ツールを付与しますが、ライブラリは「プラグイン › 内蔵」でオフになっています。画像はそのプロジェクトに保存されます",
"The model Magpie Image draws with. Agents get the Magpie Image tool from the Library; images are saved in their project": "Magpie Image が画像生成に使うモデルです。エージェントにはライブラリから Magpie Image ツールを付与します。画像はそのプロジェクトに保存されます",
"Turn on the Library": "ライブラリをオンにする",
@@ -12748,7 +12754,9 @@ const I18N = {
"How long a routing group's classifier is given to say which intent a turn is. The turn waits for it before it goes anywhere, and past it goes on without its answer. A slow classifier may need longer": "Wie lange der Klassifikator einer Routing-Gruppe Zeit hat zu sagen, welche Absicht eine Runde hat. Die Runde wartet darauf, bevor sie irgendwohin geht, und danach geht sie ohne seine Antwort weiter. Ein langsamer Klassifikator braucht vielleicht länger",
"Image generation timeout": "Zeitlimit für Bilderzeugung",
"How long a provider is given to draw an image and send it, before the request is given up. A slow image endpoint may need longer": "Wie lange ein Anbieter Zeit hat, ein Bild zu zeichnen und zu senden, bevor die Anfrage aufgegeben wird. Ein langsamer Bild-Endpunkt braucht unter Umständen länger",
- "The model Magpie Image draws with, for the agents given the Magpie Image tool; images are saved in their project": "Das Modell, mit dem Magpie Image für Agenten mit dem Magpie-Image-Tool zeichnet; Bilder werden in ihrem Projekt gespeichert",
+ "The model the Image generation MCP server below draws with, for the agents that add it; images come back in their answer": "Das Modell, mit dem der MCP-Server der Bilderzeugung unten für die Agenten zeichnet, die ihn hinzufügen; Bilder kommen in ihrer Antwort zurück",
+ "Image generation MCP server": "Bilderzeugung als MCP-Server",
+ "Agents can add image generation as an MCP server (Streamable HTTP) with generate_image, generate_video and generation_result tools, which draw with the Image generation model. An image comes back in the answer and at a URL kept for an hour, a video at a URL; nothing is saved in the agent's project. From another computer, send a gateway key as Authorization: Bearer <key>.": "Agenten können die Bilderzeugung als MCP-Server (Streamable HTTP) mit den Tools generate_image, generate_video und generation_result hinzufügen, die mit dem Modell der Bilderzeugung zeichnen. Ein Bild kommt in der Antwort zurück und liegt eine Stunde lang unter einer URL, ein Video unter einer URL; im Projekt des Agenten wird nichts gespeichert. Von einem anderen Computer aus einen Gateway-Schlüssel als Authorization: Bearer <key> senden.",
"The model Magpie Image draws with. Agents get the Magpie Image tool from the Library, which is turned off in Plugins › Built-in; images are saved in their project": "Das Modell, mit dem Magpie Image zeichnet. Agenten bekommen das Magpie-Image-Tool aus der Bibliothek, die unter Erweiterungen › Eingebaut ausgeschaltet ist; Bilder werden in ihrem Projekt gespeichert",
"The model Magpie Image draws with. Agents get the Magpie Image tool from the Library; images are saved in their project": "Das Modell, mit dem Magpie Image zeichnet. Agenten bekommen das Magpie-Image-Tool aus der Bibliothek; Bilder werden in ihrem Projekt gespeichert",
"Turn on the Library": "Bibliothek einschalten",
diff --git a/internal/gui/core_budget_test.go b/internal/gui/core_budget_test.go
index 127e7967b..b67e5d98b 100644
--- a/internal/gui/core_budget_test.go
+++ b/internal/gui/core_budget_test.go
@@ -15,8 +15,8 @@ import (
// so a page can't quietly grow back into the core; a change that needs more
// raises them and says why in its commit.
const (
- appScriptBudget = 1_321_336 // app.js: 1,198,239 after Usage left (P2); 1,218,098 with the 10-15 fixes to pages still in the core; 1,232,466 with the provider editor's Advanced fold and every dialog holding its place, the core's own (10-16); 1,250,533 with the quota refresh kept through redraws, the core's own since the tray panel draws it too, plus 1.5% (#2397); 1,269,083 with Free first's option in the provider editor's ROUTINGS, which the Routing page reads too, plus 1.5%; 1,272,726 with the Providers page drawing its last list at once (#1519), the core's page, and the Routing trace's account names read without Object.entries (#1481), plus 1.5%; a dialog growing out of the row that opened it and folding back into it, the core's own since every page's dialogs open through it (10-20); Codex's and Claude Code's own settings moved from Settings into the agent's row on the Agents page, the core's own since the tray panel and the window both draw them (10-20); 1,314,432 with Settings › Privacy's record of what was masked (its summary, the records view and the local model's controls), which belongs to Settings, the core's own, plus 1.5%; 1,336,481 with a provider's Names & levels saying Provider in model names is Settings' and opening it there (joe), and Settings' Image recognition saying where a custom provider's model is marked (nexnull), both in the core's own pages, with no room added (10-23); 1,337,633 with Settings › Models › Replies' Intent classifier timeout (#2934, reyoung), Settings' own row, drawn by waitRow, which the Image generation timeout's row now shares, with no room added; 1,337,735 with the provider editor's link naming a partner's page by its own host (siteHost) instead of usemagpie.ai's /go link, the core's own editor, with no room added (10-23); 1,338,084 with Check in now saying what the press came to (claimed +credits, already today, not available yet) where it said "refreshed" (#3326), the check-in row the tray panel draws too, the core's own, with no room added (10-23); 1,338,929 with a new partner's tile keeping a small dot until it is opened, in the add sheet, the core's own, with no room added (10-23); 1,343,324 with the sidebar's open-page row marked by one shape that moves on a spring (navTopGo), the window's own shell, which every page sits beside, with no room added (10-23); 1,343,465 with a dialog kept centred while its content comes in, until it is pressed (holdModal, every page's dialogs), the core's own, with no room added (10-24); 1,317,486 with what masking did moved out to Insights › Masking (plugins/redact), the budget lowered to it (10-24); 1,321,336 with Codex's one subagent model, its card's square saying plainly what each choice does and a row with a name in its sheet, where two squares read alike (the owner), the Agents page's, which the tray panel draws too, the core's own, with no room added (10-24)
- coreScriptBudget = 1_528_525 // every core script: 1,358,034 after Usage left; 1,380,003 with them; 1,405,937 with the fold; 1,426,102 with 10-16's core work (the GitHub list loading, the move to plugins, Settings › Long conversations), plus 1.5% (#2612); 1,447,368 with loadBack, the window's one state read per return, which every page needs and so is the core's own (1,446,845 before it, 155 bytes under), plus 1.5%; OpenCode Zen's card saying OpenCode tells no key its balance adds ~390 bytes inside it; 1,472,589 with the dialog that grows out of its row and the agents' own settings in their rows, on main's 10-20 core work (the Providers page's last list, the Routing trace's names), plus 1.5% (10-20); 1,506,192 with Settings › Privacy's record of what was masked, plus 1.5%; 1,529,393 with the Intent classifier timeout row (#2934), with no room added; 1,529,495 with the editor's partner link host, with no room added; 1,529,844 with Check in now's press summary (#3326), with no room added; 1,530,689 with the new partner tile's dot, with no room added; 1,535,084 with the sidebar's spring, with no room added; 1,549,478 with the context window card's map (tiles by tokens, the default look, the block grid a switch away) and its text view showing an inline image, in context.js, which the Routing page draws as well as Usage, so the core's own, with no room added (10-24); 1,550,768 with a usage card's refresh opening after its text (#840's card, which the tray panel draws too), the Add key spinner, and a dialog kept centred while its content comes in, with no room added (10-24); 1,524,789 with what masking did moved out to Insights › Masking, lowered to it; 1,528,525 with Codex's one subagent model, with no room added (10-24)
+ appScriptBudget = 1_321_424 // app.js: 1,198,239 after Usage left (P2); 1,218,098 with the 10-15 fixes to pages still in the core; 1,232,466 with the provider editor's Advanced fold and every dialog holding its place, the core's own (10-16); 1,250,533 with the quota refresh kept through redraws, the core's own since the tray panel draws it too, plus 1.5% (#2397); 1,269,083 with Free first's option in the provider editor's ROUTINGS, which the Routing page reads too, plus 1.5%; 1,272,726 with the Providers page drawing its last list at once (#1519), the core's page, and the Routing trace's account names read without Object.entries (#1481), plus 1.5%; a dialog growing out of the row that opened it and folding back into it, the core's own since every page's dialogs open through it (10-20); Codex's and Claude Code's own settings moved from Settings into the agent's row on the Agents page, the core's own since the tray panel and the window both draw them (10-20); 1,314,432 with Settings › Privacy's record of what was masked (its summary, the records view and the local model's controls), which belongs to Settings, the core's own, plus 1.5%; 1,336,481 with a provider's Names & levels saying Provider in model names is Settings' and opening it there (joe), and Settings' Image recognition saying where a custom provider's model is marked (nexnull), both in the core's own pages, with no room added (10-23); 1,337,633 with Settings › Models › Replies' Intent classifier timeout (#2934, reyoung), Settings' own row, drawn by waitRow, which the Image generation timeout's row now shares, with no room added; 1,337,735 with the provider editor's link naming a partner's page by its own host (siteHost) instead of usemagpie.ai's /go link, the core's own editor, with no room added (10-23); 1,338,084 with Check in now saying what the press came to (claimed +credits, already today, not available yet) where it said "refreshed" (#3326), the check-in row the tray panel draws too, the core's own, with no room added (10-23); 1,338,929 with a new partner's tile keeping a small dot until it is opened, in the add sheet, the core's own, with no room added (10-23); 1,343,324 with the sidebar's open-page row marked by one shape that moves on a spring (navTopGo), the window's own shell, which every page sits beside, with no room added (10-23); 1,343,465 with a dialog kept centred while its content comes in, until it is pressed (holdModal, every page's dialogs), the core's own, with no room added (10-24); 1,317,486 with what masking did moved out to Insights › Masking (plugins/redact), the budget lowered to it (10-24); 1,321,336 with Codex's one subagent model, its card's square saying plainly what each choice does and a row with a name in its sheet, where two squares read alike (the owner), the Agents page's, which the tray panel draws too, the core's own, with no room added (10-24); 1,321,424 with Settings › Models' Image generation MCP server row beside Image recognition's (JonahFu on X), the two drawn by one helper (mcpRow), Settings' own, with no room added (10-25)
+ coreScriptBudget = 1_528_972 // every core script: 1,358,034 after Usage left; 1,380,003 with them; 1,405,937 with the fold; 1,426,102 with 10-16's core work (the GitHub list loading, the move to plugins, Settings › Long conversations), plus 1.5% (#2612); 1,447,368 with loadBack, the window's one state read per return, which every page needs and so is the core's own (1,446,845 before it, 155 bytes under), plus 1.5%; OpenCode Zen's card saying OpenCode tells no key its balance adds ~390 bytes inside it; 1,472,589 with the dialog that grows out of its row and the agents' own settings in their rows, on main's 10-20 core work (the Providers page's last list, the Routing trace's names), plus 1.5% (10-20); 1,506,192 with Settings › Privacy's record of what was masked, plus 1.5%; 1,529,393 with the Intent classifier timeout row (#2934), with no room added; 1,529,495 with the editor's partner link host, with no room added; 1,529,844 with Check in now's press summary (#3326), with no room added; 1,530,689 with the new partner tile's dot, with no room added; 1,535,084 with the sidebar's spring, with no room added; 1,549,478 with the context window card's map (tiles by tokens, the default look, the block grid a switch away) and its text view showing an inline image, in context.js, which the Routing page draws as well as Usage, so the core's own, with no room added (10-24); 1,550,768 with a usage card's refresh opening after its text (#840's card, which the tray panel draws too), the Add key spinner, and a dialog kept centred while its content comes in, with no room added (10-24); 1,524,789 with what masking did moved out to Insights › Masking, lowered to it; 1,528,525 with Codex's one subagent model, with no room added (10-24); 1,528,972 with the Image generation MCP server row (JonahFu on X), with no room added (10-25)
)
func TestCoreScriptBudget(t *testing.T) {
diff --git a/internal/gui/tests/image-mcp.test.cjs b/internal/gui/tests/image-mcp.test.cjs
new file mode 100644
index 000000000..938ded587
--- /dev/null
+++ b/internal/gui/tests/image-mcp.test.cjs
@@ -0,0 +1,90 @@
+// Run with Node's test runner and Playwright on the module path; see README.md.
+// magpie's image generation as an MCP server (JonahFu on X): Settings ›
+// Models has an "Image generation MCP server" row, after image
+// recognition's, with the gateway's address of it, /mcp/magpie/image, to
+// copy into any agent, and says what its tools give back and that a
+// gateway key goes with it from another computer. In every language, in
+// both engines, wide and narrow.
+const assert = require("node:assert/strict");
+const fs = require("node:fs/promises");
+const path = require("node:path");
+const { test } = require("node:test");
+const { chromium, webkit } = require("playwright");
+
+const assets = path.resolve(__dirname, "../assets");
+const address = "http://127.0.0.1:3999/mcp/magpie/image";
+const tools = /generate_image.*generate_video.*generation_result.*Authorization: Bearer <key>/;
+const words = {
+ en: { name: "Image generation MCP server", sub: tools },
+ zh: { name: "生图 MCP 服务器", sub: tools },
+ "zh-TW": { name: "生圖 MCP 伺服器", sub: tools },
+ ja: { name: "画像生成 MCP サーバー", sub: tools },
+ de: { name: "Bilderzeugung als MCP-Server", sub: tools },
+};
+
+function serve(lang) {
+ const settings = { lang, theme: "light", gateway: "http://127.0.0.1:3999", port: 3999, vision: "", visionAuto: "codex/gpt-6.1-sol",
+ visionModels: [{ id: "codex/gpt-6.1-sol", name: "GPT-6.1 Sol", provider: "codex", providerName: "Codex", icon: "openai" }],
+ imageGen: "", imageGenAuto: "codex/gpt-image-2", imageGenModels: [{ id: "codex/gpt-image-2", name: "GPT Image 2", provider: "codex", providerName: "Codex", icon: "openai" }] };
+ const state = { agents: [], profiles: [], settings, clients: [] };
+ return async (r) => {
+ const url = new URL(r.request().url());
+ const json = (data) => r.fulfill({ json: data });
+ if (url.pathname === "/boot.js") return r.fulfill({ contentType: "text/javascript", body: `window.bootPrefs = {lang:"${lang}",theme:"light",web:true};` });
+ if (url.pathname === "/wails/runtime.js") return r.fulfill({ contentType: "text/javascript", body: "export const Window = {};" });
+ if (url.pathname === "/api/state") return json(state);
+ if (url.pathname === "/api/settings") return json(settings);
+ if (url.pathname === "/api/plugins") return json({ plugins: [] });
+ if (url.pathname === "/api/gateway/trace") {
+ if (url.searchParams.get("wait")) await new Promise((res) => setTimeout(res, 20e3));
+ return json({ mine: true, now: new Date().toISOString(), seq: 1, totals: { requests: 0, rerouted: 0, errors: 0 }, routes: [] });
+ }
+ if (url.pathname === "/api/groups") return json({ models: [], groups: [], pools: [] });
+ if (url.pathname === "/api/providers") return json({ providers: [], presets: [], excluded: [], gateway: { running: true, window: true, url: "http://127.0.0.1:3999" } });
+ if (url.pathname.startsWith("/api/")) return json({});
+ const file = path.join(assets, url.pathname === "/" ? "index.html" : url.pathname);
+ const contentType = { ".html": "text/html", ".js": "text/javascript", ".css": "text/css", ".svg": "image/svg+xml", ".png": "image/png" }[path.extname(file)];
+ try { await r.fulfill({ body: await fs.readFile(file), contentType }); } catch { await r.fulfill({ status: 404, body: "" }); }
+ };
+}
+
+async function open(t, engine, lang, view, width = 1100) {
+ const browser = await (engine === "webkit" ? webkit.launch() : chromium.launch({ channel: "chromium" }));
+ t.after(() => browser.close());
+ const page = await (await browser.newContext({ viewport: { width, height: 900 }, reducedMotion: "reduce" })).newPage();
+ page.setDefaultTimeout(5000);
+ const errors = [];
+ page.on("pageerror", (e) => errors.push(e.message));
+ await page.route("**/*", serve(lang));
+ await page.goto("http://magpie.test/?view=" + view);
+ return { page, errors };
+}
+
+for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium", "webkit"])) {
+ for (const lang of Object.keys(words)) {
+ const w = words[lang];
+ test(`${engine} ${lang}: Image generation gives its MCP server's address`, async (t) => {
+ for (const width of [1100, 520]) {
+ const { page, errors } = await open(t, engine, lang, "settings&tab=models", width);
+ // Settings is drawn again as its answers come in, each time with a
+ // new row: it is read in one go, never held across a redraw
+ await page.waitForFunction((a) => document.querySelector("#imagesList #imageMcpRow code")?.textContent === a, address);
+ const row = await page.evaluate(() => {
+ const r = document.querySelector("#imagesList #imageMcpRow");
+ r.scrollIntoView({ block: "nearest" });
+ return { name: r.querySelector(".name").textContent, sub: r.querySelector(".sub").textContent,
+ code: r.querySelector("code").textContent, buttons: r.querySelectorAll("button").length,
+ after: r.previousElementSibling?.id };
+ });
+ assert.equal(row.name, w.name);
+ assert.match(row.sub, w.sub);
+ assert.equal(row.code, address);
+ assert.equal(row.buttons, 1, "a copy button");
+ assert.equal(row.after, "visionMcpRow", "beside image recognition's server");
+ // nothing spills out of the window at a narrow width
+ assert(await page.evaluate(() => document.scrollingElement.scrollWidth <= innerWidth + 1), `no sideways scroll at ${width}px`);
+ assert.deepEqual(errors, []);
+ }
+ });
+ }
+}
diff --git a/internal/gui/tests/images-library-way.test.cjs b/internal/gui/tests/images-library-way.test.cjs
index b945d32bb..46ec5bb94 100644
--- a/internal/gui/tests/images-library-way.test.cjs
+++ b/internal/gui/tests/images-library-way.test.cjs
@@ -173,7 +173,7 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",
assert.equal(await row(page).locator("button.image-gen-tool").count(), 0);
const sub = await row(page).locator(".who > .sub").first().textContent();
assert.equal(sub, opts.gateway
- ? w("The model Magpie Image draws with, for the agents given the Magpie Image tool; images are saved in their project")
+ ? w("The model the Image generation MCP server below draws with, for the agents that add it; images come back in their answer")
: w("Agents given Magpie Image can't generate images or videos: the tool says it is off"));
assert.deepEqual(page.errors, []);
await page.context().close();
diff --git a/internal/gui/tests/search-api.test.cjs b/internal/gui/tests/search-api.test.cjs
index be1169e92..ff3fbc9b4 100644
--- a/internal/gui/tests/search-api.test.cjs
+++ b/internal/gui/tests/search-api.test.cjs
@@ -95,6 +95,22 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",
for (let i = 0; ; i++) {
try { await head.scrollIntoViewIfNeeded({ timeout: 1000 }); break; } catch (e) { if (i >= 4) throw e; }
}
+ // the clicks below are at the row's buttons' places, so the whole row
+ // is brought on screen, by the wheel: magpie puts back a scroll the
+ // reader didn't make, and a row only partly shown is left where it is
+ // by scrollIntoViewIfNeeded; Settings' own landing can still put the
+ // view back, so the row is on screen twice, a second apart
+ for (let i = 0, seen = 0; ; i++) {
+ const below = await head.evaluate((e) => Math.ceil(e.getBoundingClientRect().bottom - innerHeight + 24));
+ if (below <= 0 && ++seen === 2) break;
+ assert(i < 12, `the search row comes on screen: ${below}px below`);
+ if (below > 0) {
+ seen = 0;
+ await page.mouse.move(width / 2, 700);
+ await page.mouse.wheel(0, below);
+ }
+ await page.waitForTimeout(below > 0 ? 200 : 1000);
+ }
assert.equal(await list.locator("xpath=preceding-sibling::div[1]").textContent(), w.head);
assert.equal(await head.locator(".name").innerText(), w.name);
assert((await head.locator(".sub").innerText()).includes(w.none));
diff --git a/internal/imagemcp/imagemcp.go b/internal/imagemcp/imagemcp.go
index 0499fbf4f..dea67fac0 100644
--- a/internal/imagemcp/imagemcp.go
+++ b/internal/imagemcp/imagemcp.go
@@ -38,12 +38,20 @@ const (
VideoFolder = "generated-videos"
)
+// The tools and their arguments are the gateway's image MCP server's too
+// (gateway.ImageMCPPath), but where to save, which only this server can.
const (
- tool = "generate_image"
- videoTool = "generate_video"
- resultTool = "generation_result"
+ tool = gateway.GenerateImageTool
+ videoTool = gateway.GenerateVideoTool
+ resultTool = gateway.GenerationResultTool
)
+// withPath is props with path, where to save, said as what.
+func withPath(props map[string]any, what string) map[string]any {
+ props["path"] = map[string]any{"type": "string", "description": what}
+ return props
+}
+
// answerWithin is how long a tool call waits for its image or video before
// it answers that it is still being made, with the job's id to ask after
// it by. Agents give a tool call as little as 30 s (#575: ZCode's "Tool
@@ -71,14 +79,8 @@ var videoSchema = map[string]any{
"/ unless path says where) and its path is returned; when it is not ready within about 20 seconds, the answer is an id to wait on with " + resultTool + " instead. Describe the subject, the motion and the camera: what moves, how, and what the shot is.",
"inputSchema": map[string]any{
"type": "object",
- "properties": map[string]any{
- "prompt": map[string]any{"type": "string", "description": "What happens in the video: the subject, its motion, the camera."},
- "path": map[string]any{"type": "string", "description": "Where to save it: a file (e.g. assets/intro.mp4) or a folder, relative to the project or absolute. Never written over."},
- "seconds": map[string]any{"type": "integer", "minimum": 1, "maximum": 15, "description": "Length in whole seconds (1-15). Default: the model's (about 8)."},
- "size": map[string]any{"type": "string", "description": "An aspect ratio (16:9, 9:16, 1:1, 4:3, 3:4, 3:2, 2:3) or WIDTHxHEIGHT (1280x720; its shorter side picks 480p, 720p or 1080p). Default: the model's."},
- "image": map[string]any{"type": "string", "description": "An image to animate, the video's first frame: a file path (relative to the project or absolute) or an http(s) URL."},
- "reference_images": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "Images the video's subjects are drawn from: file paths or http(s) URLs."},
- },
+ "properties": withPath(gateway.VideoToolProps("a file path (relative to the project or absolute) or an http(s) URL", "file paths or http(s) URLs"),
+ "Where to save it: a file (e.g. assets/intro.mp4) or a folder, relative to the project or absolute. Never written over."),
"required": []string{"prompt"},
},
}
@@ -88,11 +90,9 @@ var resultSchema = map[string]any{
"description": "Wait for an image or video that " + tool + " or " + videoTool + " said is still being made, by the id it gave, and get where it was saved. " +
"It answers within about " + strconv.Itoa(int(answerWithin/time.Second)) + " seconds: if it is still not ready, call this again with the same id (don't generate it again).",
"inputSchema": map[string]any{
- "type": "object",
- "properties": map[string]any{
- "id": map[string]any{"type": "string", "description": "The id " + tool + " or " + videoTool + " answered with."},
- },
- "required": []string{"id"},
+ "type": "object",
+ "properties": gateway.ResultToolProps(),
+ "required": []string{"id"},
},
}
@@ -103,15 +103,8 @@ var schema = map[string]any{
"reference it from code or docs by that path. Write a detailed prompt: subject, style, composition, colours, any text to render exactly.",
"inputSchema": map[string]any{
"type": "object",
- "properties": map[string]any{
- "prompt": map[string]any{"type": "string", "description": "What to draw, or how to change the reference images."},
- "path": map[string]any{"type": "string", "description": "Where to save it: a file (e.g. assets/hero.png) or a folder, relative to the project or absolute. With n > 1 a number is added to the name."},
- "size": map[string]any{"type": "string", "description": "WIDTHxHEIGHT (1024x1024, 1536x1024, 1024x1536) or an aspect ratio (16:9). Default: the model's."},
- "n": map[string]any{"type": "integer", "minimum": 1, "maximum": 4, "description": "How many images (default 1)."},
- "reference_images": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "Images to edit or draw from: file paths or http(s) URLs."},
- "quality": map[string]any{"type": "string", "description": "low, medium, high or auto, for models that take it."},
- "background": map[string]any{"type": "string", "description": "transparent, opaque or auto, for models that take it."},
- },
+ "properties": withPath(gateway.ImageToolProps("file paths or http(s) URLs"),
+ "Where to save it: a file (e.g. assets/hero.png) or a folder, relative to the project or absolute. With n > 1 a number is added to the name."),
"required": []string{"prompt"},
},
}